PHP data scraping and multi_cURL number of processes limit
<p><br>
I'm working on a script that will scrape data from another site using cURL and DOMDocument.
I'll post 3 part of code to better explain my script (sorry if it's a big long), but I would check if my logic in building script is correct (I've never worked with scrape and multi_cURL) and I'm not a professional programmer.</p>

<p><strong>As a first step</strong> I've made a single process running and it worked.
Code was this:</p>

<pre><code>$urlCurl = '[url to scrap]';

$options = Array(
            CURLOPT_RETURNTRANSFER =&gt; TRUE,  // Setting cURL's option to return the webpage data
            CURLOPT_FOLLOWLOCATION =&gt; TRUE,  // Setting cURL to follow 'location' HTTP headers
            CURLOPT_AUTOREFERER =&gt; TRUE, // Automatically set the referer where following 'location' HTTP headers
            CURLOPT_CONNECTTIMEOUT =&gt; 300,   // Setting the amount of time (in seconds) before the request times out
            CURLOPT_TIMEOUT =&gt; 300,  // Setting the maximum amount of time for cURL to execute queries
            CURLOPT_MAXREDIRS =&gt; 10, // Setting the maximum number of redirections to follow
            CURLOPT_USERAGENT =&gt; "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8",  // Setting the useragent
            CURLOPT_URL =&gt; $urlCurl, // Setting cURL's URL option with the $url variable passed into the function
        );

$chCurl = curl_init($urlCurl);
curl_setopt_array($chCurl, $options);   // Setting cURL's options using the previously assigned array data in $options
$resultChCurl = curl_exec($chCurl);
$html = $resultChCurl;

$dom = new DOMDocument();
$html = $dom-&gt;loadHTML($html);

$dom-&gt;preserveWhiteSpace = false;

//$result = array();

$tables = $dom-&gt;getElementsByTagName('table');
foreach($tables as $table) {
    if ($table-&gt;hasAttribute('class')) {
        $rows = $dom-&gt;getElementsByTagName("tr");
            foreach($rows as $row) {

                $cols = $row-&gt;getElementsByTagName('td'); 
                $aref = $row-&gt;getElementsByTagName('a');

                foreach($aref as $profile) {
                    if (($cols-&gt;item(0)-&gt;nodeValue != "Seller") and ($cols-&gt;item(1)-&gt;nodeValue != "Ratio") and ($cols-&gt;item(2)-&gt;nodeValue != "Amount")) {
                        $seller =  $cols-&gt;item(0)-&gt;nodeValue; 
                        $sellerID = intval(preg_replace('/[^0-9]+/', '', ($profile-&gt;getAttribute( 'href' ))), 10);
                        $sellerType = sellerTypeExtract($profile-&gt;getAttribute( 'href' ));
                        $currencysell = preg_replace('/[.0-9]/','',($cols-&gt;item(1)-&gt;nodeValue)); 
                        $amount =  floatval($cols-&gt;item(1)-&gt;nodeValue);
                        $rate = rateExtractor($cols-&gt;item(2)-&gt;nodeValue);
                        $currencybought = preg_replace('/[.0-9'.$currencysell.'=]/','',($cols-&gt;item(2)-&gt;nodeValue));
                        $marketLink = $urlCurl;
                        $server = "servername";
                        $tenGoldref = tenGoldref($rate);
                    }

                    $result[] = array (
                        'seller' =&gt; $seller,
                        'sellerprofile'=&gt; $sellerID,
                        'sellertype' =&gt; $sellerType,
                        'currencysold' =&gt; $currencysell,
                        'amount'=&gt; $amount,
                        'rate'=&gt; $rate,
                        'currencybought'=&gt; $currencybought,
                        'marketLink' =&gt; $marketLink,
                        'server' =&gt; $server,
                        'tengold' =&gt; $tenGoldref,
                        'timeMoment' =&gt; $dayTime,
                        'date' =&gt; $date,
                        'hour' =&gt; $hour
                    );
                }
            } 
        }
    }}

    print_r($result);
</code></pre>

<p><strong>Than I've tried to implement multi_cURL</strong> in order to save time and I've started with this simple script.</p>

<pre><code>function urlBuilder() {

$countryList = array();
for ($i = 1; $i &lt;= 50; $i++) {
    echo "&lt;br&gt;".$i."&lt;br&gt;";
    $origin = "[VARIABLE URL TO SCRAP]";
    if (empty($countryList)) {
        $countryList = array($origin);
    } else {
        (array_push($countryList,$origin));
    }
}
return $countryList;
}

function start() {

   $fullArr = array();

   $nodes = urlBuilder();
   $node_count = count($nodes);

   $curl_arr = array();
   $master = curl_multi_init();

   for($i = 0; $i &lt; $node_count; $i++)
      {
        $url = $nodes[$i];
        $curl_arr[$i] = curl_init($url);
        curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
        curl_multi_add_handle($master, $curl_arr[$i]);
      } 
   do {
        curl_multi_exec($master,$running);
      } while($running &gt; 0);
   echo "results: ";
   for($i = 0; $i &lt; $node_count; $i++)
     {
        $results = curl_multi_getcontent  ( $curl_arr[$i]  );
        $infos = scrape($results,$nodes[$i],$dayTime,$date,$hour);

        if (empty($fullArr)) {
           $fullArr = array($infos);
        } else {
        (array_push($fullArr,$infos));
        }
   }

   echo "&lt;br&gt;&lt;br&gt;*******************NOW THERE SHOULD BE FULL ARRAY*************************&lt;br&gt;";
   echo 'done';
   echo "&lt;br&gt;";
   print_r($fullArr);
   echo "&lt;br&gt;&lt;br&gt;*******************END FULL ARRAY*************************&lt;br&gt;";
</code></pre>

<p>but the first problem I met was that with 155 parallel connections the last 30/40 send back "NULL" data (not everytime in the 90% of tests).
<strong>So the first question is: does the script return NULL data because there are too many connections?</strong></p>

<p>I thought that, that's why I've tried to implement this tutorial (<a href="https://www.flynsarmy.com/2013/08/a-more-efficient-utf-8-curl_multi/" rel="nofollow" title="link to tutorial">link to tutorial</a>) in my code, in order to have a good multi connection handler, and the final code was this:</p>

<pre><code> function curl_multi_download(array $urls, callable $callback, array $custom_options = array())
 {
     $time_start_insert = microtime(true);

     // make sure the rolling window isn't greater than the # of urls
     $rolling_window = 50;
     $rolling_window = (sizeof($urls) &lt; $rolling_window) ? sizeof($urls) : $rolling_window;

     $master = curl_multi_init();
     $curl_arr = array();
     $options = array(
         CURLOPT_RETURNTRANSFER =&gt; true,
         CURLOPT_FOLLOWLOCATION =&gt; true,
         CURLOPT_MAXREDIRS =&gt; 5,
     ) + $custom_options;

     // start the first batch of requests
     for ( $i = 0; $i &lt; $rolling_window; $i++ )
     {
        $ch = curl_init();
        $options[CURLOPT_URL] = $urls[$i];
        curl_setopt_array($ch, $options);
        curl_multi_add_handle($master, $ch);
    }
  do
    {
  while(($execrun = curl_multi_exec($master, $running)) == CURLM_CALL_MULTI_PERFORM);
  if($execrun != CURLM_OK)
        break;
  // a request was just completed -- find out which one
  while( $done = curl_multi_info_read($master) )
       {
        $info = curl_getinfo($done['handle']);

        // request successful.  process output using the callback function.
        $output = curl_multi_getcontent($done['handle']);
        $extract = call_user_func_array($callback, array($info, $output));

        if (empty($fullArr)) {
            $fullArr = array($extract);
        } else {
            (array_push($fullArr,$extract));
        }

        echo "&lt;br&gt;&lt;br&gt;*******************NOW THERE SHOULD BE FULL ARRAY*************************&lt;br&gt;";
        echo 'done';
        echo "&lt;br&gt;";
        print_r($fullArr);
        echo "&lt;br&gt;&lt;br&gt;*******************END FULL ARRAY*************************&lt;br&gt;";
        $time_end_insert = microtime(true);
        //dividing with 60 will give the execution time in minutes other wise seconds
        $execution_time_insert = ($time_end_insert - $time_start_insert)/60;
        //execution time of the script
        echo '&lt;br&gt;&lt;br&gt;&lt;p class="finalTime"&gt;++++++++ total execution time: '.round($execution_time_insert,2,PHP_ROUND_HALF_DOWN).' minutes +++++++++++&lt;/p&gt;&lt;br&gt;';

        if ( isset($urls[$i+1]) )
        {
            // start a new request (it's important to do this before removing the old one)
            $ch = curl_init();
            $options[CURLOPT_URL] = $urls[$i++];  // increment i
            curl_setopt_array($ch, $options);
            curl_multi_add_handle($master, $ch);
        }

        // remove the curl handle that just completed
        curl_multi_remove_handle($master, $done['handle']);
      }
   } while ($running);

   curl_multi_close($master);
   return true;
}

function curl_multi_getcontent_utf8( $ch )
{
   $data = curl_multi_getcontent( $ch );
   if ( !is_string($data) )
      return $data;

   unset($charset);
   $content_type = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);

   /* 1: HTTP Content-Type: header */
   preg_match( '@([\w/+]+)(;\s*charset=(\S+))?@i', $content_type, $matches );
   if ( isset( $matches[3] ) )
       $charset = $matches[3];

   /* 2: &lt;meta&gt; element in the page */
    if ( !isset($charset) )
    {
        preg_match( '@&lt;meta\s+http-equiv="Content-Type"\s+content="([\w/]+)(;\s*charset=([^\s"]+))?@i', $data, $matches );
        if ( isset( $matches[3] ) )
           $charset = $matches[3];
    }

    /* 3: &lt;xml&gt; element in the page */
    if ( !isset($charset) )
    {
    preg_match( '@&lt;\?xml.+encoding="([^\s"]+)@si', $data, $matches );
    if ( isset( $matches[1] ) )
        $charset = $matches[1];
    }

    /* 4: PHP's heuristic detection */
   if ( !isset($charset) )
   {
      $encoding = mb_detect_encoding($data);
      if ($encoding)
        $charset = $encoding;
   }

   /* 5: Default for HTML */
   if ( !isset($charset) )
   {
       if (strstr($content_type, "text/html") === 0)
          $charset = "ISO 8859-1";
   }

   /* Convert it if it is anything but UTF-8 */
   /* You can change "UTF-8"  to "UTF-8//IGNORE" to
   ignore conversion errors and still output something reasonable */
   if ( isset($charset) &amp;&amp; strtoupper($charset) != "UTF-8" )
      $data = iconv($charset, 'UTF-8', $data);

   return $data;
}

curl_multi_download(array('[LINKS TO SCRAPE]'), 'process_response');

function process_response( $info, $response ) 
{

if ( $info['http_code'] != 200 )
{
    echo "Error retrieving URL " . $info['url'] . "&lt;br/&gt;";
    return;
};
//var_dump($info);
//var_dump($response);
    $fullArr = array();
    $infos = scrape($response,$dayTime,$date,$hour);
    return $infos;
}

function curl_download( $url )
{
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
  $data = self::curl_exec($ch);
  curl_close($ch);

  return $data;
}

function scrape($data,$dayTime,$date,$hour) {

  $html = $data;
  $dom = new DOMDocument();
  $html = $dom-&gt;loadHTML($html);

  $dom-&gt;preserveWhiteSpace = false;

  $result = array();

  $tables = $dom-&gt;getElementsByTagName('table');
  foreach($tables as $table) {
     if ($table-&gt;hasAttribute('class')) {
        $rows = $dom-&gt;getElementsByTagName("tr");
            foreach($rows as $row) {

                $cols = $row-&gt;getElementsByTagName('td'); 
                $aref = $row-&gt;getElementsByTagName('a');

                foreach($aref as $profile) {
                    if (($cols-&gt;item(0)-&gt;nodeValue != "Seller") and ($cols-&gt;item(1)-&gt;nodeValue != "Ratio") and ($cols-&gt;item(2)-&gt;nodeValue != "Amount")) {
                        $seller =  $cols-&gt;item(0)-&gt;nodeValue; 
                        $sellerID = intval(preg_replace('/[^0-9]+/', '', ($profile-&gt;getAttribute( 'href' ))), 10);
                        $sellerType = sellerTypeExtract($profile-&gt;getAttribute( 'href' ));
                        $currencysell = preg_replace('/[.0-9]/','',($cols-&gt;item(1)-&gt;nodeValue)); 
                        $amount =  floatval($cols-&gt;item(1)-&gt;nodeValue);
                        $rate = rateExtractor($cols-&gt;item(2)-&gt;nodeValue);
                        $currencybought = preg_replace('/[.0-9'.$currencysell.'=]/','',($cols-&gt;item(2)-&gt;nodeValue));
                        //$marketLink = $nodes;
                        $server = "servername";
                        $tenGoldref = tenGoldref($rate);
                    }

                    $result = array (
                        'seller' =&gt; $seller,
                        'sellerprofile'=&gt; $sellerID,
                        'sellertype' =&gt; $sellerType,
                        'currencysold' =&gt; $currencysell,
                        'amount'=&gt; $amount,
                        'rate'=&gt; $rate,
                        'currencybought'=&gt; $currencybought,
                        //'marketLink' =&gt; $marketLink,
                        'server' =&gt; $server,
                        'tengold' =&gt; $tenGoldref,
                        'timeMoment' =&gt; $dayTime,
                        'date' =&gt; $date,
                        'hour' =&gt; $hour
                    );
                }
            } 
        }
    }       
    return $result;
 }
</code></pre>

<p>and so what happened?
Every scrape of every page, <strong>in spite of returning me a lot of data (like 10/12 records for each page scraped in final array), script started returning me only 1 record (first record) for each page, even if I didn't changed anything in scrape function. Do you know why?</strong>.</p>

<p>More than this do you have any multi_cURL handler to suggest to try, maybe better than the one I've choosed, or this one is good?</p>

<p>I hope this wasn't a too much noob question.
Sorry for long post.</p>

<p>Thx for your time.</p>