import pandas as pd
with open(r'data.json') as f:
df = pd.read_json(f, encoding='utf-8')
Im getting "Could not reserve memory block" error. Json has 300 mb size, is there any limit for reserving memory for running program in python? I have 8 GB RAM on PC, using windows 10
loading of json file into df
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2018.1.4\helpers\pydev\pydev_run_in_console.py", line 52, in run_file
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2018.1.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/Beorn/PycharmProjects/project_0/projekt/test.py", line 7, in <module>
df = pd.read_json(f, encoding='utf-8')
File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 422, in read_json
result = json_reader.read()
File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 529, in read
obj = self._get_object_parser(self.data)
File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 546, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 638, in parse
self._parse_no_numpy()
File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 853, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Could not reserve memory block
PyDev console: starting.
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)] on win32
So after reading plenty of posts and solutions I decided to just reduce my file size by getting rid of uselles data. Maybe you find this usefull. Btw. I read somewhere that u need at least x25 more memory than your json file has, so in my case i needed more than 8Gb.