Intermittent ELB 4XX errors on service with tight timout window
<p>This issue has been very difficult to investigate and diagnose.  We have an amazon ALB serving traffic to multiple internal tornado servers (in docker containers managed by ECS).  The response time of these servers is very consistent, and very small: usually 4-5 ms.</p>

<p>We then have a PHP server which uses this tornado based server via the ALB.  It makes curl requests to the ALB address, but given the nature of this environment, we require really tight tolerances on the connection timeout and request timeout.  50 ms and 100 ms, respectively.  Due to these tight windows, we also retry 3 times if a timeout does occur.</p>

<p>Given that the tornado based response time is only 5 ms, I would expect these tolerances to be more than enough.  Additionally, we have scaled up the EC2 instances so that CPU never goes above ~25%, so it shouldn't be a CPU limiting thing.</p>

<p>Unfortunately, we are seeing intermittent issues where the ELB 4XX errors for the ALB jump significantly, until roughly 3% of the traffic is returning timing out on all 4 requests/retries.  We also don't see any indication of issues on the tornado server.</p>

<p>I'm wondering if anybody else has run into issue like this, or if anybody has suggestions for how to investigate.  PHP is reporting timeouts, the ALB is reporting 4XX issues, and the tornado server is continuing to show 5 ms responses and doesn't indicate anything is wrong.</p>

<p>Here is the curl request from the PHP side:</p>

<pre><code>$curl = curl_init();

curl_setopt_array($curl, array(
    CURLOPT_RETURNTRANSFER =&gt; 1,
    CURLOPT_URL =&gt; $url,
    CURLOPT_USERAGENT =&gt; 'web backend',
    CURLOPT_CONNECTTIMEOUT_MS =&gt; 50,
    CURLOPT_TIMEOUT_MS =&gt; 100,
    CURLOPT_POST =&gt; 1,
    CURLOPT_POSTFIELDS =&gt; $jsonContent,
    CURLOPT_HTTPHEADER =&gt; array(
        'Content-Type: application/json',
        'Content-Length: ' . strlen($jsonContent))));
));

$response = curl_exec($curl);

$retry = 0;
while(curl_errno($curl) != 0 &amp;&amp; $retry &lt; 3) {
    $response = curl_exec($curl);
    $retry++;
}

// log error
</code></pre>