Python 3 unicode encode error

Question

I'm using glob.glob to get a list of files from a directory input. When trying to open said files, Python fights me back with this error:

UnicodeEncodeError: 'charmap' codec can't encode character '\xf8' in position 18: character maps to < undefined >

By defining a string variable first, I can do this:

filePath = r"C:\Users\Jørgen\Tables\\"

Is there some way to get the 'r' encoding for a variable?

EDIT:

import glob

di = r"C:\Users\Jørgen\Tables\\"

def main():
    fileList = getAllFileURLsInDirectory(di)
    print(fileList)

def getAllFileURLsInDirectory(directory):
    return glob.glob(directory + '*.xls*')

There is a lot more code, but this problem stops the process.

Answer 1

Please also see:

Convert python filenames to unicode and Listing chinese filenames in directory with python

You can tell Python to explicitly handle strings as unicode -- but you have to maintain that from the first string onward.

In this case passing a u'somepath' to os.walk.

Answer 2

Independently on whether you use the raw string literal or a normal string literal, Python interpreter must know the source code encoding. It seems you use some 8-bit encoding, not the UTF-8. Therefore you have to add the line like

# -*- coding: cp1252 -*-

at the beginning of the file (or using another encoding used for the source files). It need not to be the first line, but it usually is the first or second (the first should contain #!python3 for the script used on Windows).

Anyway, it is usually better not to use non ASCII characters in the file/directory names.

You can also use normal slashes in the path (the same way as in Unix-based systems). Also, have a look at os.path.join when you need to compose the paths.

Updated

The problem is probably not where you search it for. My guess is that the error manifests only when you want to display the resulting list via print. This is usually because the console by default uses non-unicode encoding that is not capable to display the character. Try the chcp command without arguments in your cmd window.

You can modify the print command in your main() function to convert the string representation to the ASCII one that can always be displayed:

print(ascii(fileList))