How does long-polling work in Tornado?

Question

In Tornado's chat demo, it has a method like this:

@tornado.web.asynchronous
def post(self):
    cursor = self.get_argument("cursor", None)
    global_message_buffer.wait_for_messages(self.on_new_messages,
                                            cursor=cursor)

I'm fairly new to this long polling thing, and I don't really understand exactly how the threading stuff works, though it states:

By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections...

My theory was that by making a simple app:

import tornado.ioloop
import tornado.web
import time

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        print("Start request")
        time.sleep(4)
        print("Okay done now")
        self.write("Howdy howdy howdy")
        self.finish()

application =  tornado.web.Application([
    (r'/', MainHandler),
])

That if I made two requests in a row (i.e. I opened two browser windows and quickly refreshed both) I would see this:

Start request
Start request
Okay done now
Okay done now

Instead, I see

Start request
Okay done now
Start request
Okay done now

Which leads me to believe that it is, in fact, blocking in this case. Why is it that my code is blocking, and how do I get some code to do what I expect? I get the same output on Windows 7 with a core i7, and a linux Mint 13 box with I think two cores.

Edit:

I found one method - if someone can provide a method that works cross-platform (I'm not too worried about performance, only that it's non-blocking), I'll accept that answer.

Answer 1

Since Tornado 5.0, asyncio is enabled automatically, so pretty much just changing time.sleep(4) to await asyncio.sleep(4) and @tornado.web.asynchronous def get(self): to async def get(self): solves the problem.

Example:

import tornado.ioloop
import tornado.web
import asyncio

class MainHandler(tornado.web.RequestHandler):
    async def get(self):
        print("Start request")
        await asyncio.sleep(4)
        print("Okay done now")
        self.write("Howdy howdy howdy")
        self.finish()

app =  tornado.web.Application([
    (r'/', MainHandler),
])
app.listen(8888)
tornado.ioloop.IOLoop.current().start()

Output:

Start request
Start request
Okay done now
Okay done now

Sources:

• Tornado on asyncio
• asyncio usage example

Answer 2

The problem with the code in original question is that when you call time.sleep(4) you are effectively blocking the execution of event loop for 4 seconds. And accepted answer doesn't solve the problem either (IMHO).

Asynchronous serving in Tornado works on trust. Tornado will call your functions whenever something happens, but it trusts you that you will return control to it as soon as possible. If you block with time.sleep() then this trust is breached - Tornado can't handle new connections.

Using multiple threads only hides the mistake; running Tornado with thousands of threads (so you can serve 1000s of connections simultaneously) would be very inefficient. The appropriate way is running a single thread which only blocks inside Tornado (on select or whatever Tornado's way of listening for events is) - not on your code (to be exact: never on your code).

The proper solution is to just return from get(self) right before time.sleep() (without calling self.finish()), like this:

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        print("Starting")

You must of course remember that this request is still open and call write() and finish() on it later.

I suggest you take a look at chat demo. Once you strip out the authentication you get a very nice example of async long polling server.

Answer 3

The right way to convert your test app into a form that won't block the IOLoop is like this:

from tornado.ioloop import IOLoop
import tornado.web
from tornado import gen
import time

@gen.coroutine
def async_sleep(timeout):
    """ Sleep without blocking the IOLoop. """
    yield gen.Task(IOLoop.instance().add_timeout, time.time() + timeout)

class MainHandler(tornado.web.RequestHandler):
    @gen.coroutine
    def get(self):
        print("Start request")
        yield async_sleep(4)
        print("Okay done now")
        self.write("Howdy howdy howdy")
        self.finish()

if __name__ == "__main__":
    application =  tornado.web.Application([
        (r'/', MainHandler),
    ])
    application.listen(8888)
    IOLoop.instance().start()

The difference is replacing the call to time.sleep with one which won't block the IOLoop. Tornado is designed to handle lots of concurrent I/O without needing multiple threads/subprocesses, but it will still block if you use synchronous APIs. In order for your long-polling solution to handle concurrency the way you'd like, you have to make sure that no long-running calls block.