Issue with C++ Multithreaded file read running on virtual machines
<p>This is more of a conceptual question. A legacy code that I'm working with has a C++ thread implementation to emulate parallel file reading. The way it handles the multithreaded file read is by streaming only a block of data (let's say <code>2^20 bytes</code>) from the file to an <code>char* array</code> then multiple threads reading dedicated section of this array. When a thread complete it waits for all the other threads to complete (i.e., using a <code>condition_variable</code> shared on a <code>unique_lock</code>) and the last thread to complete, loads the next block of data from file into the array and continue the process until whole file is read. The reason for parallel reading is that often these files are over <code>1 GB</code> and very time consuming otherwise. The speed up is about <code>6x</code> or more doing this.</p>

<p>I have observed some strange behavior. When the code runs in a physical machine (even if it is remotely connected) the multithreaded file read always work correctly. However, if it is run on a virtual machine, sometimes it is missing data after the read. I have debugged with each thread reading only one line and still sometimes the results are incorrect. This showed the reason for incorrect results is not incorrect reading but missing entire lines from the input file (a <code>.txt</code>). I can reproduce the faulty results more often if I increase the number of threads (higher the thread count more often it fails) and by increasing the file size (i.e., if the file size is greater than the size of the <code>char* array</code>). This lead me to believe the issue is not in the multithreaded file read but something to do with the run-time environment (physical vs virtual machine). </p>

<p>I'm just thinking out loud and want to see if someone have experienced similar behavior or if any insights as to why such behavior could occur. I'm investigating of any <code>buffer overflow</code> issues but so far didn't find any. </p>

<p><strong>EDIT:</strong> <em>Adding more information regarding the environment</em></p>

<p>I'm using <code>Visual Studios 2013</code> so utilizing the <code>MS Visual C++</code> compiler and all the physical and virtual machines are <code>Windows Server 2012 R2</code> or <code>Windows Server 2008 R2</code> with <code>64-bit</code> and <code>x64-based processors</code> Minimum memory available is <code>32GB</code> and could go upt to <code>120GB</code>. Servers use different versions of <code>Intel Xeon E5</code> CPUs.  </p>