Apache not releasing threads with low volume of requests
<p>On one of our servers, we are running in to a little curious behavior when one of our deployed apps runs an update against our API.  The gist is that it downloads an XML file that, among other things, contains a file list.  It then proceeds to retrieve the files listed.  The files are served by a PHP process that consumes the file ID, looks it up, and reads the file out to the client if the file exists.</p>

<p>Initially, due to low concurrency (we are running at 2, right now) and file size, the files will come in at approximately 1 per second.  However, as the process draws on, our developers are indicating that download times will raise to over 30 seconds per image.</p>

<p>When looking at the Munin charts, the raise in download times coincides with a rise in Apache Processes (and, thus, memory consumption) around the same time.</p>

<p><a href="https://i.stack.imgur.com/JBP76.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JBP76.png" alt="Apache Processes"></a></p>

<p>The curious part is, the actual volume of Apache Accesses, over the course of the update, remains very low.</p>

<p><a href="https://i.stack.imgur.com/oQT11.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oQT11.png" alt="Apache Accesses"></a></p>

<p>While (unless I am reading this chart wrong) the number of active connections continues to grow.</p>

<p><a href="https://i.stack.imgur.com/k3G7t.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/k3G7t.png" alt="Firewall Connections"></a></p>

<p>It seems odd to us that the processes count just keeps rising like this, and it seems like completed connections are not being dropped.  I would assume that as requests were completed, that the connection would terminate, and the Apache worker would be re-used on a subsequent connection.  However it does not look like this is occurring...  Further, you can see that the number of open connections and active Apache processes does not significantly change under "higher" access load conditions.</p>

<p>The script, as it currently stands, amounts to:</p>

<pre><code>if (
    $file = File::get($_GET['file_id']) &amp;&amp;
    !$file-&gt;is_locked() &amp;&amp; $file-&gt;has_member_access($current_member)
){
    /* Stuff */

    header('Cache-Control: public');
    header('Last-Modified: ' . $gmt_modify_time );
    header('Content-Length: ' . $file_size);
    header('Content-Disposition: attachment; filename="' . $file_name . '";');
    header('Content-type: ' . $file_type);
    header('Content-Transfer-Encoding: binary');


    @readfile("$file_path") or die("File not found."); 

}
</code></pre>

<p>So, essentially, this is hit once per second, over a period of time, until the Apache processes stack up to the point that it is hit once every 30 seconds.  The tip of the charts below represent termination of the update, and show that the processes drop off slowly as well...</p>

<p>One last thing, and a bonus question, is "why do connections stay open for so long after they have been, I would guess, closed?"  You can see in <code>ipconntrack</code>, a plateau of ~275 connections sticking around until Wednesday.  That plateau is nearly a week long, and coincides with Established Connections through Firewall.</p>

<p><a href="https://i.stack.imgur.com/GhxkY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GhxkY.png" alt="Connections Week"></a>
<a href="https://i.stack.imgur.com/6x27S.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6x27S.png" alt="Firewall Week"></a></p>

<p>All very curious, and frustrating...</p>