PS get-content high memory usage - Is there a more efficient way to filter a file?
<p>I am using get-content to read a largish file (252 MB), but when I use get-content to read it, the powershell process proceeds to consume almost 10 GB of memory. Is this normal behavior?</p>
<p>The array has just shy of 6 million items. It doesn't seem to be remotely in line with the amount of memory being used.</p>
<p>Maybe I'm just going about this the wrong way entirely.</p>
<p>I want to write the line that matches a string and the subsequent line to a new text file.</p>
<pre><code>$mytext = get-content $inpath
$search = &quot;*tacos*&quot;
$myindex = 0..($mytext.count - 1) | Where {$mytext[$_] -like $search}
$outtext = @()
foreach ($i in $myindex){
    $outtext = $outtext + $mytext[$i] + $mytext[$i+1]
    }
$outtext | out-file -filepath $outpath
</code></pre>
<h1>Performance Testing Results</h1>
<p>I took a performance sample for different scripts based on different answers here.</p>
<p><strong>My original script</strong></p>
<p>(highly sensitive to the number of lines that get written out)</p>
<ul>
<li>10k lines - 1.8s</li>
<li>100k lines - 38s</li>
<li>100k lines - 21s (when search string occurs rarely)</li>
<li>5000k lines - Too long to measure (aborted after several hours)</li>
</ul>
<p><strong>Select-String with no Get-Content (adapted from <a href="https://stackoverflow.com/a/38873560/598943">whatever</a>)</strong></p>
<p><code>Select-String -path $inpath -pattern $search -Context 0,1 -SimpleMatch | Out-File $outpath</code></p>
<ul>
<li>10k lines - 1.2s</li>
<li>100k lines -  4s</li>
<li><strong>1000k lines - 107s</strong></li>
</ul>
<p>Note processing speed only grows by a factor of ~4 for a 10x increase in input. The more data you try to process at once, the better this solution becomes relative to the others.</p>
<p><strong>Eliminating the array resize (from Mathias)</strong></p>
<ul>
<li>10k lines - 2.0s</li>
<li>100k lines - 25s</li>
<li>1000k lines - 1533s (using 1.7GB memory, the same as running gc outside of the script on 1000k lines)</li>
</ul>
<p><strong>Using the pipeline (from Chris Dent)</strong></p>
<ul>
<li>100k lines - 26s</li>
</ul>