Reading and processing large volatile memory file byte by byte
<p>I'm currently using python to work on memory dumps created by apps like belkasoft, ftk imager and so on, they are usually of the extension .mem or .dmp, and of the form:</p>

<pre><code>53 FF 00 F0 53 FF 00 
</code></pre>

<p>I'd like to visualize these data let's say use a heatmap or use curves maybe trying to visualize each byte. It would be 2147483648 bytes in case of 2GB File.
How would you approach reading and processing this kind of large files >= 2GB </p>

<p>I've been experimenting with something like:</p>

<pre><code>with open("File.mem",'rb') as file:
    byte = file.read(1)
    while byte:
         Do something
</code></pre>

<p>and managed to do some calculation but its painfully slow
I also tried reading the file line by line which was fast but again reading the bytes of each line and doing some conversion was also painfully slow.
I've read also read some about numpy loadtxt, but didn't experiment much with it thought i'd ask here first.  </p>

<p>Any ideas that might suit this scenario and make it more efficient? </p>

<p>Thanks a lot</p>