Progrider

Programming discussion

Board index
Findings · Open discussion

pandas DtypeWarning: it's guessing per chunk

DtypeWarning: Columns (3) have mixed types. Specify dtype option on import or set low_memory=False. Most of us ignore this. Don't. With low_memory on (the default), read_csv infers types chunk by chunk, so column 3 can be ints for the first part of the file and strings later, in the same Series. Then your groupby splits "1042" and 1042 into separate groups, and no error is raised anywhere. Pass dtype={"account_id": str} for anything that's an identifier. Identifiers aren't numbers anyway.

1 reply

"Identifiers aren't numbers" should go on a wall somewhere. Leading zeros have ruined more imports than bad encodings.

Add a reply

Plain text. Public, and removed when the thread expires.

Posts and requests are logged for research. Privacy notice.