How to get the total number of records processed by Spark Streaming?
<p>Does anyone know how does Spark compute its number of records (I think it is the same as the number of events in a batch), as displayed here?</p>

<p><a href="https://i.stack.imgur.com/UDOkD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UDOkD.png" alt="enter image description here"></a></p>

<p>I'm trying to figure out how I can get this value remotely (REST-API does not exist for Streaming option in the UI). </p>

<p>Basically what I'm trying to do it to get the total number of records processed by my application. I need this information for the web portal. </p>

<p>I tried to count the <code>Records</code> for each stage, but it gave me completely different number as it is at the picture above. Each stage contain the infomation about its records. As shown here</p>

<p><a href="https://i.stack.imgur.com/OVl98.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OVl98.png" alt="enter image description here"></a></p>

<p>I'm using this short python script to count the "inputRecords", from each stage. This is the source code:</p>

<pre><code>import json, requests, urllib
print "Get stages script started!"
#URL REST-API
url = 'http://10.16.31.211:4040/api/v1/applications/app-20161104125052-0052/stages/'
response = urllib.urlopen(url)
data = json.loads(response.read())

stages = []
print len(data)
inputCounter = 0
for item in data:
        stages.append(item["stageId"])
        inputCounter += item["inputRecords"]
print "Records processed: " + str(inputCounter)
</code></pre>

<p>If I understood it correctly: Each <code>Batch</code> has one <code>Job</code>, and each <code>Job</code> has multiple <code>Stages</code>, these <code>Stages</code> have multiple <code>Tasks</code>. </p>

<p>So for me it made sense to count the input for each <code>Stage</code>.</p>