文本文件到内存的Python的负载2GB(Python load 2GB of text file

2019-07-30 01:40发布

在Python 2.7，当我加载从2.5GB的一个文本文件到内存中的所有数据，这样更快的处理：

>>> f = open('dump.xml','r')
>>> dump = f.read()

我得到了以下错误：

Python(62813) malloc: *** mmap(size=140521659486208) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

为什么Python的尝试分配140521659486208字节的内存为2563749237字节的数据？如何修复代码，使其加载所有字节？

我在周围3GB内存免费。该文件是一个XML维基转储。

Answer 1:

如果你使用的mmap ，你就可以将整个文件立即加载到内存中。

import mmap

with open('dump.xml', 'rb') as f:
  # Size 0 will read the ENTIRE file into memory!
  m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) #File is open read-only

  # Proceed with your code here -- note the file is already in memory
  # so "readine" here will be as fast as could be
  data = m.readline()
  while data:
    # Do stuff
    data = m.readline()