Dask read_csv fails where pandas doesn't

问题:

Trying to use dask's read_csv on file where pandas's read_csv like this

dd.read_csv('data/ecommerce-new.csv')

fails with the following error:

pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at line 2

The file is csv file of scraped data using scrapy with two columns, one with the url and the other with the html(which is stored multiline using " as delimiter char). Being actually parsed by pandas means it should be actually well-formatted.

html,url
https://google.com,"<a href=""link"">
</a>"

Making the sample argument big enough to load the entire file in memory seems to work, which makes me believe it actually fails when trying to infer the datatypes(also there's this issue which should have been solved https://github.com/dask/dask/issues/1284)

Has anyone encountered this problem before? Is there a fix/workaround?

EDIT: Apparently this is a known problem with dask's read_csv if the file contains a newline character between quotes. A solution I found was to simply read it all in memory:

dd.from_pandas(pd.read_csv(input_file), chunksize=25)

This works, but at the cost of parallelism. Any other solution?

回答1:

For people coming here in 2020, the dd.read_csv works directly for newlines inside quotes. It has been fixed. Update to the latest version of Dask (2.18.1) to get these benefits.

For people who want to use an older version for some reason, as suggested by @mdurant you might wanna pass blocksize=None to dd.read_csv which will be at a cost of parallel loading.