How to intercept partial updates to accumulators on driver?
<p>Spark 1.5.1 + Java 1.8</p>

<p>We are using spark to upload a solid ammount of records to database.</p>

<p>The Action code looks like this:</p>

<pre><code>rdd.foreachPartition(new VoidFunction&lt;Iterator&lt;T&gt;&gt;() {

     @Override
     public void call(Iterator&lt;T&gt; iter) {
          //while there are more records perform the following every 1000 records
          //int[] recoords = statement.executeBatch();
          //accumulator.add(recoords.length);
     }
     // ...
} 
</code></pre>

<p>On the driver node there is a thread that monitors the accumulator value. However the value does not get updated. It does get updated only once, by the time applications ends. Even if accumulators used lazy value setting, it should be updated correctly since I am reading the value periodically in the driver node thread.</p>

<p>Am I using the accumulator incorrectly? Is there anyway I can more continuously monitor progress from my workers?</p>