Facebook JSON badly encoded

问题:

I downloaded my Facebook messenger data (in your Facebook account, go to settings, then to Your Facebook information, then Download your information, then create a file with at least the Messages box checked) to do some cool statistics

However there is a small problem with encoding. I'm not sure, but it looks like Facebook used bad encoding for this data. When I open it with text editor I see something like this: Rados\u00c5\u0082aw. When I try to open it with python (UTF-8) I get RadosÅ\x82aw. However I should get: Radosław.

My python script:

text = open(os.path.join(subdir, file), encoding='utf-8')
conversations.append(json.load(text))

I tried a few most common encodings. Example data is:

{
  "sender_name": "Rados\u00c5\u0082aw",
  "timestamp": 1524558089,
  "content": "No to trzeba ostatnie treningi zrobi\u00c4\u0087 xD",
  "type": "Generic"
}

回答1:

I can indeed confirm that the Facebook download data is incorrectly encoded; a Mojibake. The original data is UTF-8 encoded but was decoded as Latin -1 instead. I’ll make sure to file a bug report.

In the meantime, you can repair the damage in two ways:

1. Decode the data as JSON, then re-encode any strings as Latin-1, decode again as UTF-8:

   >>> import json
   >>> data = r'"Rados\u00c5\u0082aw"'
   >>> json.loads(data).encode('latin1').decode('utf8')
   'Radosław'
   

2. Load the data as binary, replace all \u00hh sequences with the byte the last two hex digits represent, decode as UTF-8 and then decode as JSON:

   import re
   from functools import partial
   
   fix_mojibake_escapes = partial(
        re.compile(rb'\\u00([\da-f]{2})').sub,
        lambda m: bytes.fromhex(m.group(1).decode()))
   
   with open(os.path.join(subdir, file), 'rb') as binary_data:
       repaired = fix_mojibake_escapes(binary_data.read())
   data = json.loads(repaired.decode('utf8'))
   

   From your sample data this produces:

   {'content': 'No to trzeba ostatnie treningi zrobić xD',
    'sender_name': 'Radosław',
    'timestamp': 1524558089,
    'type': 'Generic'}
   

回答2:

My solution for parsing objects use parse_hook callback on load/loads function:

import json


def parse_obj(dct):
    for key in dct:
        dct[key] = dct[key].encode('latin_1').decode('utf-8')
        pass
    return dct


data = '{"msg": "Ahoj sv\u00c4\u009bte"}'

# String
json.loads(data)  
# Out: {'msg': 'Ahoj svÄ\x9bte'}
json.loads(data, object_hook=parse_obj)  
# Out: {'msg': 'Ahoj světe'}

# File
with open('/path/to/file.json') as f:
     json.load(f, object_hook=parse_obj)
     # Out: {'msg': 'Ahoj světe'}
     pass

Update:

Solution for parsing list with strings does not working. So here is updated solution:

import json


def parse_obj(obj):
    for key in obj:
        if type(obj[key]) is str:
            obj[key] = obj[key].encode('latin_1').decode('utf-8')
        elif type(obj[key]) is list:
            obj[key] = list(map(lambda x: x if type(x) != str else x.encode('latin_1').decode('utf-8'), obj[key]))
        pass
    return obj