Why do long HTTP round trip-times stall my Tornado AsyncHttpClient?
<p>I'm using Tornado to send requests in rapid, periodic succession (every 0.1s or even 0.01s) to a server. For this, I'm using <code>AsyncHttpClient.fetch</code> with a callback to handle the response.
Here's a very simple code to show what I mean:</p>

<pre><code>from functools import partial
from tornado import gen, locks, httpclient
from datetime import timedelta, datetime

    # usually many of these running on the same thread, maybe requesting the same server
 @gen.coroutine
 def send_request(url, interval):
    wakeup_condition = locks.Condition()
    #using this to allow requests to send immediately
    http_client = httpclient.AsyncHTTPClient(max_clients=1000) 

    for i in range(300):
        req_time = datetime.now()

        current_callback = partial(handle_response, req_time)
        http_client.fetch(url, current_callback, method='GET')
        yield wakeup_condition.wait(timeout=timedelta(seconds=interval))

def handle_response(req_time,  response):
    resp_time = datetime.now()
    write_to_log(req_time, resp_time, resp_time - req_time) #opens the log and writes to it
</code></pre>

<p>When I was testing it against a local server, it was working fine, the requests were being sent on time, the round trip time was obviously minimal.
However, when I test it against a remote server, with larger round trip times (especially for higher request loads), the request timing gets messed up by multiple seconds: The period of wait between each request becomes much larger than the desired period. </p>

<p>How come? I thought the async code wouldn't be affected by the roundtrip time since it isn't blocking while waiting for the response. Is there any known solution to this?</p>