Python 3 UnicodeDecodeError: 'ascii' codec

Question

I'm implementing this notebook on Windows with Python 3.5.3 and got the follow error on load_vectors() call. I've tried different solutions posted but none worked.

<ipython-input-86-dd4c123b0494> in load_vectors(loc)
      1 def load_vectors(loc):
      2     return (load_array(loc+'.dat'),
----> 3         pickle.load(open(loc+'_words.pkl','rb')),
      4         pickle.load(open(loc+'_idx.pkl','rb')))

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Answer 1

You should probably give encoding for pickle.load(f, encoding='latin1'), but please make sure all the characters in your file will follow the encoding.

By default, your pickle code is trying to decode the file with 'ASCII' which fails. Instead you can explicitly tell which one to use. See this from Documentation.

If latin1 doesn't solve, try with encoding='bytes' and then decode all the keys and values later on.

Answer 2

I solved this issue by copying and pasting the entire csv file into text and reading it with:

with open(self.path + "/review_collection.txt", "r", encoding="utf-8") as f:
    read = f.read().splitlines()
    for row in read:
        print(row)