PHP File Writing (fwrite / file_put_contents) speed/optimization
<p>So, i have a database with big data. The data to use is currently about 2,6 GB.</p>

<p>All the data need to be written to a text file for later use in another scripts.</p>

<p>The data is being limited per file and splitted in multiple parts. 100 results per file (around 37MB each file). Thats about 71 files.</p>

<p>The data is json data that is being serialized and then encrypted with openssl.</p>

<p>The data is correctly being written to the files, untill the max execution time is reached after 240 seconds. That's after about 20 files...</p>

<p>Well, i can just extend that time, but thats not the problem.</p>

<p>The problem is the following:</p>

<pre><code>Writing file 1-6:   +/- 5 seconds
Writing file 7-8:   +/- 7 seconds
Writing file 9-11:  +/- 12 seconds
Writing file 12-14: +/- 17 seconds
Writing file 14-16: +/- 20 seconds
Writing file 16-18: +/- 23 seconds
Writing file 19-20: +/- 27 seconds

Note: time is needed time per file
</code></pre>

<p>In other words, with every file im writing, the writing time per file goes significantly up, what causes the script to be slow offcourse.</p>

<p>The structure of the script is a bit like this:</p>

<pre><code>$needed_files = count needed files/parts

for ($part=1; $part&lt;=$needed_files; $part++) { // Loop throught parts
    $query &gt; mysqli select data
    $data &gt; json_encode &gt; serialize &gt; openssl_encrypyt
    file_put_contents($filename.$part, $data, LOCK_EX);
}
</code></pre>

<p><strong>WORKING CODE AFTER HELP</strong></p>

<pre><code>$notchDetails = mysqli_query($conn, "SELECT * FROM notches WHERE projectid = ".$projectid."");

$rec_count = 0;
$limit = 100;
$part = 1;

while ($notch = mysqli_fetch_assoc($notchDetails)) {

    $data1[] = $notch;
    $rec_count++;

    if ($rec_count &gt;= $limit) {

        $data = json_encode($data1);
        $data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
        $filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
        file_put_contents($filename, $data, LOCK_EX);

        $part++;
        $rec_count = 0;
        $data = $data1 = "";

    }

}
if ($data1 != "") {
    $data = json_encode($data1);
    $data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
    $filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
    file_put_contents($filename, $data, LOCK_EX);
}

mysqli_free_result($notchDetails);
</code></pre>