Changing the Delimiter in a large CSV file using Powershell
<p>I am in need of a way to change the delimiter in a CSV file from a comma to a pipe.  Because of the size of the CSV files (~750 Mb to several Gb), using Import-CSV and/or Get-Content is not an option.  What I'm using (and what works, albeit slowly) is the following code:</p>

<pre><code>$reader = New-Object Microsoft.VisualBasic.FileIO.TextFieldParser $source
$reader.SetDelimiters(",")

While(!$reader.EndOfData)
{   
    $line = $reader.ReadFields()
    $details = [ordered]@{
                            "Plugin ID" = $line[0]
                            CVE = $line[1]
                            CVSS = $line[2]
                            Risk = $line[3]     
                         }                        
    $export = New-Object PSObject -Property $details
    $export | Export-Csv -Append -Delimiter "|" -Force -NoTypeInformation -Path "C:\MyFolder\Delimiter Change.csv"    
}
</code></pre>

<p>This little loop took nearly 2 minutes to process a 20 Mb file.  Scaling up at this speed would mean over an hour for the smallest CSV file I'm currently working with.  </p>

<p>I've tried this as well:</p>

<pre><code>While(!$reader.EndOfData)
{   
    $line = $reader.ReadFields()  

    $details = [ordered]@{
                             # Same data as before
                         }

    $export.Add($details) | Out-Null        
}

$export | Export-Csv -Append -Delimiter "|" -Force -NoTypeInformation -Path "C:\MyFolder\Delimiter Change.csv"
</code></pre>

<p>This is MUCH FASTER but doesn't provide the right information in the new CSV.  Instead I get rows and rows of this:</p>

<pre><code>"Count"|"IsReadOnly"|"Keys"|"Values"|"IsFixedSize"|"SyncRoot"|"IsSynchronized"
"13"|"False"|"System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection"|"System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection"|"False"|"System.Object"|"False"
"13"|"False"|"System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection"|"System.Collections.Specialized.OrderedDictionary+OrderedDictionaryKeyValueCollection"|"False"|"System.Object"|"False"
</code></pre>

<p>So, two questions:</p>

<p>1) Can the first block of code be made faster?
2) How can I unwrap the arraylist in the second example to get to the actual data?</p>

<p>EDIT: Sample data found here - <a href="http://pastebin.com/6L98jGNg" rel="nofollow">http://pastebin.com/6L98jGNg</a></p>