Cross read parquet files between R and Python

问题:

We have generated a parquet files, one in Dask (Python) and another with R Drill (using the Sergeant packet ). They use a different implementations of parquet see my other parquet question

We are not able to cross read the files (the python can't read the R file and vice versa).
When reading the Python parquet file in the R environment we receive the following error: system error: Illegalstatexception: UTF8 can only annotate binary filed .
When reading the R/Drill parquet file in Dask we get an FileNotFoundError: [Error 2] no such file or directory ...\_metadata (which is self explanatory).
What are the options to cross read parquet files between R and Python?

Any insights would be appreciated.

回答1:

To read drill-like parquet data-sets with fastparquet/dask, you need to pas a list of the filenames, e.g.,

files = glob.glob('mydata/*/*.parquet')
df = dd.read_parquet(files)

The error from going in the other direction might be a bug, or (gathering from your other question), may indicate that you used fixed-length strings, but drill/R doesn't support them.