Asynchronous file writing possible in python?

问题:

Is there an easy way write to a file asynchronously in Python?

I know the file io that comes with Python is blocking; which is fine in most cases. For this particular case, I need writes not to block the application at all, or at least as minimally as possible.

回答1:

Twisted has non-blocking writes on file descriptors. If you're writing async code, I'd expect you to be using twisted, anyway. :)

回答2:

As I understand things, asynchronous I/O is not quite the same as non-blocking I/O.

In the case of non-blocking I/O, once a file descriptor is setup to be "non-blocking", a read() system call (for instance) will return EWOULDBLOCK (or EAGAIN) if the read operation would block the calling process in order to complete the operation. The system calls select(), poll(), epoll(), etc. are provided so that the process can ask the OS to be told when one or more file descriptors become available for performing some I/O operation.

Asynchronous I/O operates by queuing a request for I/O to the file descriptor, tracked independently of the calling process. For a file descriptor that supports asynchronous I/O (raw disk devcies typically), a process can call aio_read() (for instance) to request a number of bytes be read from the file descriptor. The system call returns immediately, whether or not the I/O has completed. Some time later, the process then polls the operating system for the completion of the I/O (that is, buffer is filled with data).

A process (single-threaded) that only performs non-blocking I/O will be able to read or write from one file descriptor that is ready for I/O when another is not ready. But the process must still synchronously issue the system calls to perform the I/O to all the ready file descriptors. Whereas, in the asynchronous I/O case, the process is just checking for the completion of the I/O (buffer filled with data). With asynchronous I/O, the OS has the freedom to operate in parallel as much as possible to service the I/O, if it so chooses.

With that, are there any wrappers for the POSIX aio_read/write etc. system calls for Python?

回答3:

I'm developing aio.h bidings to python: pyaio

It runs on linux only..

回答4:

Python 3 seems to have such functionality. See PEP 3116.

回答5:

You can try to use Thread:

from threading import Thread

for file in list_file:
     tr = Thread(target=file.write, args=(data,))
     tr.start()

This is more or less a pseudocode, but i hope you'd get the idea. Be aware the threads here are left open.

In my experience it worked well, although interpreter continue to work some time while the main script have finished (need to use join()), so the speed gain is so big than it seems