Getting Dill Memory Error when loading serialized

问题:

I am getting a dill/pickle memory error when loading a serialized object file. I am not quite sure what is happening and I am unsure on how to fix it. When I call:

stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)

The code traces to the train_batch_iterator function in which it loads a serialized object and trains the classifier with the data within the object. This is the code:

def train_batch_iterator(clf, tte):
plot_data = [] # initialize plot data array
for file in glob.glob('./SerializedData/Batch8172015_19999/*'):
    with open(file, 'rb') as stream:
        minibatch_train = dill.load(stream)
        clf.partial_fit(minibatch_train.data[1], minibatch_train.target, 
                         classes=np.array([11, 111]))
        tte += len(minibatch_train.target)
        plot_data.append((test_batch_iterator(clf), tte))
return plot_data

Here is the error:

Traceback (most recent call last):
  File "LArSoftSGD-version2.0.py", line 154, in <module>
    stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS)
  File "LArSoftSGD-version2.0.py", line 118, in train_batch_iterator
    minibatch_train = dill.load(stream)
  File "/home/jdoe/.local/lib/python3.4/site-packages/dill/dill.py", line 199, in load
    obj = pik.load()
  File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1038, in load
    dispatch[key[0]](self)
  File "/home/jdoe/.local/lib/python3.4/pickle.py", line 1184, in load_binbytes
    self.append(self.read(len))
  File "/home/jdoe/.local/lib/python3.4/pickle.py", line 237, in read
    return self.file_read(n) 
MemoryError

I have no idea what could be going wrong. The error seems to be in the line minibatch_train = dill.load(stream) and the only thing I can think of is that the serialized data file is too large, however the file is exactly 1161 MB which doesn't seem to big/big enough to cause a memory error. Does anybody know what might be going wrong?