Why do long HTTP round trip-times stall my Tornado

Question

I'm using Tornado to send requests in rapid, periodic succession (every 0.1s or even 0.01s) to a server. For this, I'm using AsyncHttpClient.fetch with a callback to handle the response. Here's a very simple code to show what I mean:

from functools import partial
from tornado import gen, locks, httpclient
from datetime import timedelta, datetime

    # usually many of these running on the same thread, maybe requesting the same server
 @gen.coroutine
 def send_request(url, interval):
    wakeup_condition = locks.Condition()
    #using this to allow requests to send immediately
    http_client = httpclient.AsyncHTTPClient(max_clients=1000) 

    for i in range(300):
        req_time = datetime.now()

        current_callback = partial(handle_response, req_time)
        http_client.fetch(url, current_callback, method='GET')
        yield wakeup_condition.wait(timeout=timedelta(seconds=interval))

def handle_response(req_time,  response):
    resp_time = datetime.now()
    write_to_log(req_time, resp_time, resp_time - req_time) #opens the log and writes to it

When I was testing it against a local server, it was working fine, the requests were being sent on time, the round trip time was obviously minimal. However, when I test it against a remote server, with larger round trip times (especially for higher request loads), the request timing gets messed up by multiple seconds: The period of wait between each request becomes much larger than the desired period.

How come? I thought the async code wouldn't be affected by the roundtrip time since it isn't blocking while waiting for the response. Is there any known solution to this?

Answer 1

After some tinkering and tcpdumping, I've concluded that two things were really slowing down my coroutine. With these two corrected stalling has gone down enormously drastically and the timeout in yield wakeup_condition.wait(timeout=timedelta(seconds=interval)) is much better respected:

1. The computer I'm running on doesn't seem to be caching DNS, which for AsyncHTTPClient seem to be a blocking network call. As such every coroutine sending requests has the added time to wait for the DNS to resolve. Tornado docs say:

tornado.httpclient in the default configuration blocks on DNS resolution but not on other network access (to mitigate this use ThreadedResolver or a tornado.curl_httpclient with a properly-configured build of libcurl).

...and in the AsynHTTPClient docs

To select curl_httpclient, call AsyncHTTPClient.configure at startup:

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

I ended up implementing my own thread which resolves and caches DNS, however, and that resolved the issue by issuing the request directly to the IP address.

2. The URL I was using was HTTPS, changing to a HTTP url improved performance. For my use case that's not always possible, but it's good to be able to localize part of the issue