Dataflow OutOfMemoryError while reading small tables from BigQuery
<p>We have a <a href="http://i.stack.imgur.com/C7Jyb.png" rel="nofollow">pipeline</a> reading data from BigQuery and processing historical data for various calendar years. It <strong>fails with OutOfMemoryError</strong> errors <strong>if the input data is small</strong> (~500MB)</p>

<p>On startup it reads from BigQuery about 10.000 elements/sec, after short time it slows down to hundreds elements/s then it hangs completely.</p>

<p>Observing 'Elements Added' on the next processing step (BQImportAndCompute), the value increases and then decreases again. That looks to me like some already loaded data is dropped and then loaded again.</p>

<p>Stackdriver Logging console contains errors with various stack traces that contain java.lang.OutOfMemoryError, for example: </p>

<blockquote>
  <p>Error reporting workitem progress update to Dataflow service:</p>
</blockquote>

<pre><code>"java.lang.OutOfMemoryError: Java heap space
    at com.google.cloud.dataflow.sdk.runners.worker.BigQueryAvroReader$BigQueryAvroFileIterator.getProgress(BigQueryAvroReader.java:145)
    at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation$SynchronizedReaderIterator.setProgressFromIteratorConcurrent(ReadOperation.java:397)
    at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation$SynchronizedReaderIterator.setProgressFromIterator(ReadOperation.java:389)
    at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation$1.run(ReadOperation.java:206)
</code></pre>

<p>I would suspect that there is a problem with topology of the pipe, but running the same pipeline</p>

<ol>
<li><strong>locally</strong> with DirectPipelineRunner <strong>works</strong> fine</li>
<li><strong>in cloud</strong> with DataflowPipelineRunner <strong>on large dataset</strong> (5GB, for another year) <strong>works</strong> fine</li>
</ol>

<p>I assume problem is how Dataflow parallelizes and distributes work in the pipeline. Are there any possibilities to inspect or influence it?</p>