pandas: oserror with accent/special character in f

Question

I am trying to use pandas.read_csv to get data from some .csv files. This works fine as long as there is no accent (e.g. ä,é,ü) in the file name or file path. As soon as I use a file name such as düm1.csvI get the following error: OSError: Initializing from file failed. My code is:

dum1 = pd.read_csv(r"C:\Users\MyName\Desktop\dumm12\düm1.csv", sep = ";", decimal = ",", encoding = "utf-8")

I am using pandas 0.20.1 and python 3.6.0. I have found that this has been an issue in previous versions but I thought it had been resolved. Any ideas on how to fix this? I also found this: https://github.com/pandas-dev/pandas/issues/15086

output of pd.show_versions():

INSTALLED VERSIONS commit: None python: 3.6.0.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel byteorder: little LC_ALL: None LANG: en LOCALE: None.None

pandas: 0.20.1 pytest: 3.0.5 pip: 9.0.1 setuptools: 27.2.0 Cython: None numpy: 1.11.3 scipy: 0.18.1 xarray: None IPython: 5.2.2 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: 3.2.2 numexpr: 2.6.2 feather: None matplotlib: 2.0.0 openpyxl: 2.4.1 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.3 bs4: 4.5.3 html5lib: 0.999 sqlalchemy: 1.1.5 pymysql: None psycopg2: None jinja2: 2.9.5 s3fs: None pandas_gbq: None pandas_datareader: None

Answer 1

The issue hasn't been resolved till now. Wait till a PR. Or try it with Python 2.7 I guess that might work

Answer 2

I tested the name in creating a fake file 'düm1.csv'.

when I run :

df = pd.read_csv('düm1.csv',sep=';')

I haven't an OSError and the file is open in my Ipython.

   Unnamed: 0  test1  test2  test3  tes4
0         NaN    1.0    2.0    3.0   4.0
1         NaN    NaN    NaN    NaN   NaN
2         NaN    NaN    NaN    NaN   NaN
3         NaN    NaN    NaN    NaN   NaN
4         NaN    NaN    NaN    NaN   NaN

Have you tried without encoding ? Without accent ?

C.

Answer 3

I had a similar problem. It's look like the problem occurs with pandas.read_csv with Python 3.6 in a Windows system.

Python 3.6 change Windows filesystem encoding from "mbcs" to "UTF-8". See Python PEP 529. You can use the command sys.getfilesystemencoding() to get the current file system encoding

I get two solutions around this:

1.- Use this code to change all the app to works with the prior Python <= 3.5 encoding ("mbcs")

import sys
sys._enablelegacywindowsfsencoding()

2.- Pass a file pointer to the pandas.read_csv

with open("C:\Users\MyName\Desktop\dumm12\düm1.csv", 'r') as fp:
        dum1 = pd.read_csv(fp, sep = ";", decimal = ",", encoding = "utf-8")

You can see this post: pandas.read_csv can't import file with accent mark in path