How do I add a custom monitoring feature in my Spark application?
<p>I am developing a Spark application. The application takes data from Kafka queue and processes that data. After processing it stores data in Hbase table.</p>

<p>Now I want to monitor some of the performance attributed such as,</p>

<ol>
<li>Total count of input and output records.(Not all records will be persisted to Hbase, some of the data may be filtered out in processing)</li>
<li>Average processing time per message</li>
<li>Average time taken to persist the messages.</li>
</ol>

<p>I need to collect this information and send it to a different Kafka queue for monitoring.</p>

<p>Considering that the monitoring should not incur a significant delay in the processing.</p>

<p>Please suggest some ideas for this.</p>

<p>Thanks. </p>