asynchronous programming in python

Question

Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?

Answer 1

You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/2010/08/delegate-in-pythonpython-asynchronous-programming.html

import time, random, sys
from delegate import *

def proc(a):
    time.sleep(random.random())
    return str(a)

def proc_callback(handle, args=None):
    ret = d.end(handle)

d = Delegate()
d.init(2) # number of workers

handle = d.begin(proc, '12345', proc_callback, 'test')
sys.stdin.readline()

d.free()

Answer 2

You may well want to checkout the Twisted library for Python. They provide many useful tools.

1. A little primer
2. Defer and Related stuff

Answer 3

Good news everyone!

Python 3.4 would include brand new ambitious asynchronous programming implementation!

It is currently called tulip and already has an active following.

As described in PEP 3153: Asynchronous IO support and PEP 3156: Asynchronous IO Support Rebooted:

People who want to write asynchronous code in Python right now have a few options:

• asyncore and asynchat;
• something bespoke, most likely based on the select module;
• using a third party library, such as Twisted or gevent.

Unfortunately, each of these options has its downsides, which this PEP tries to address.

Despite having been part of the Python standard library for a long time, the asyncore module suffers from fundamental flaws following from an inflexible API that does not stand up to the expectations of a modern asynchronous networking module.

Moreover, its approach is too simplistic to provide developers with all the tools they need in order to fully exploit the potential of asynchronous networking.

The most popular solution right now used in production involves the use of third party libraries. These often provide satisfactory solutions, but there is a lack of compatibility between these libraries, which tends to make codebases very tightly coupled to the library they use.

This current lack of portability between different asynchronous IO libraries causes a lot of duplicated effort for third party library developers. A sufficiently powerful abstraction could mean that asynchronous code gets written once, but used everywhere.

Here is the brief overview of it's abilities.

Answer 4

What you describe (the main program flow resuming immediately while another function executes) is not what's normally called "asynchronous" (AKA "event-driven") programming, but rather "multitasking" (AKA "multithreading" or "multiprocessing"). You can get what you described with the standard library modules threading and multiprocessing (the latter allows actual concurrent execution on multi-core machines).

Asynchronous (event-driven) programming is supported in the standard Python library in the asyncore and asynchat modules, which are very oriented to networking tasks (indeed they internally use the select module, which, on Windows, only supports sockets -- though on Unixy OSs it can also support any file descriptor).

For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.

Answer 5

Take a look here:

Asynchronous Programming in Python

An Introduction to Asynchronous Programming and Twisted

Worth checking out:

asyncio (previously Tulip) has been checked into the Python default branch

Edited on 14-Mar-2018

Today Python has asyncIO — Asynchronous I/O, event loop, coroutines and tasks built in.

Description taken from the link above:

The asyncIO module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives. Here is a more detailed list of the package contents:

1. a pluggable event loop with various system-specific implementations;
2. transport and protocol abstractions (similar to those in Twisted);
3. concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls, and others (some may be system-dependent);
4. a Future class that mimics the one in the concurrent.futures module, but adapted for use with the event loop;
5. coroutines and tasks based on yield from (PEP 380), to help write concurrent code in a sequential fashion;
6. cancellation support for Futures and coroutines;
7. synchronization primitives for use between coroutines in a single thread, mimicking those in the threading module;
8. an interface for passing work off to a threadpool, for times when you absolutely, positively have to use a library that makes blocking I/O calls.

Asynchronous programming is more complex than classical “sequential” programming: see the Develop with asyncio page which lists common traps and explains how to avoid them. Enable the debug mode during development to detect common issues.

Also worth checking out:

A guide to asynchronous programming in Python with asyncIO

Answer 6

The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you're already using something else that provides its own loop.

Here is a contrived example that would demonstrate using the threading module:

from threading import Thread

def background_stuff():
  while True:
    print "I am doing some stuff"

t = Thread(target=background_stuff)
t.start()

# Continue doing some other stuff now

However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.

The threading module provides many such primitives for you to use, if you know how to use them.