How to overcome server load issues when running multiple cron jobs simultaneously?
<p>I've got a site that displays data from a game server. The game has different &quot;domains&quot; (which are actually just separate servers) that the users play on.</p>
<p>Right now, I've got 14 <code>cron</code> jobs running at different intervals of the hour every 6 hours. All 14 files that are run are pretty much the same, and <strong>each takes around 75 minutes</strong> ( an hour and 15 minutes ) to complete it's run.</p>
<p>I had thought about just using 1 file run from <code>cron</code> and looping through each server, but this would just cause that one file run <strong>for 18 hours</strong> or so. <strong>My current VPS is set to only allow 1 <code>vCPU</code></strong>, so I'm trying to accomplish things and stay within my allotted server load.</p>
<p>Seeing that the site needs to have <strong>updated data available every 6 hours</strong>, this isn't doable.</p>
<p>I started looking into message queues and passing some information to a background process that will perform the work in question. I started off trying to use <strong><code>resque</code></strong> and <strong><code>php-resque</code></strong>, but my background worker died as soon as it was started. So, I moved on to <strong><code>ZeroMQ</code></strong>, which seems to be more what I need, anyway.</p>
<p>I've set up ZMQ via Composer, and everything during the installation went fine. In my worker script (which will be called by cron every 6 hours), I've got:</p>
<pre><code>$dataContext = new ZMQContext();
$dataDispatch = new ZMQSocket($dataContext, ZMQ::SOCKET_PUSH);
$dataDispatch-&gt;bind(&quot;tcp://*:50557&quot;);

$dataDispatch-&gt;send(0);

foreach($filesToUse as $filePath){
    $dataDispatch-&gt;send($filePath);
    sleep(1);
}

$filesToUse = array();
$blockDirs = array_filter(glob('mapBlocks/*'), 'is_dir');
foreach($blockDirs as $k =&gt; $blockDir){
    $files = glob($rootPath.$blockDir.'/*.json');
    $key = array_rand($files);
    $filesToUse[] = $files[$key];
}

$mapContext = new ZMQContext();
$mapDispatch = new ZMQSocket($mapContext, ZMQ::SOCKET_PUSH);
$mapDispatch-&gt;bind(&quot;tcp://*:50558&quot;);

$mapDispatch-&gt;send(0);

foreach($filesToUse as $blockPath){
    $mapDispatch-&gt;send($blockPath);
    sleep(1);
}
</code></pre>
<p><code>$filesToUse</code> is an array of files submitted by users that contain information to be used in querying the game server. As you can see, I'm looping through the array and sending each file to the <code>ZeroMQ</code> <strong><code>listener</code></strong> file, which contains:</p>
<pre><code>$startTime = time();

$context = new ZMQContext();

$receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
$receiver-&gt;connect(&quot;tcp://*:50557&quot;);

$sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
$sender-&gt;connect(&quot;tcp://*:50559&quot;);

while(true){
    $file = $receiver-&gt;recv();

    // -------------------------------------------------- do all work here
    // ... ~ 75:00 [min] DATA PROCESSING SECTION foreach .recv()-ed WORK-UNIT
    // ----------------------------------------------------------------------

    $endTime = time();
    $totalTime = $endTime - $startTime;
    $sender-&gt;send('Processing of domain '.listener::$domain.' competed on '.date('M-j-y', $endTime).' in '.$totalTime.' seconds.');
}
</code></pre>
<p>Then, in the <strong>final</strong> <strong><code>listener</code></strong> file:</p>
<pre><code>$context = new ZMQContext();
$receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
$receiver-&gt;bind(&quot;tcp://*:50559&quot;);

while(true){
    $log = fopen($rootPath.'logs/sink_'.date('F-jS-Y_h-i-A').'.txt', 'a');
    fwrite($log, $receiver-&gt;recv());
    fclose($log);
}
</code></pre>
<p>When the worker script is run from <code>cron</code>, I get no confirmation text in my log.</p>
<p><del><strong><code>Q1)</code></strong> is this <strong>the most efficient way</strong> to do what I'm trying to?</del><br>
<strong><code>Q2)</code></strong> am I trying to use or implement <strong><code>ZeroMQ</code></strong> incorrectly, here?</p>
<p><del>And, as it would seem, using <code>cron</code> to call 14 files simultaneously is causing the load to far exceed the allotment. I know I could probably just set the jobs to run at different times throughout the day, but if at all possible, I would like to keep all updates on the same schedule.</del></p>
<hr />
<h2>UPDATE:</h2>
<p>I have since gone ahead and upgraded my VPS to 2 CPU cores, so the load aspect of the question isn't really all that relevant anymore.</p>
<p>The code above has also been changed to the current setup.</p>
<p>I am, after the code-update, getting an email from <code>cron</code> now with the error:</p>
<blockquote>
<p><strong><code>Fatal error: Uncaught exception</code></strong> <code>'ZMQSocketException' with message 'Failed to bind the ZMQ: Address already in use'</code></p>
</blockquote>