Spark Streaming: Using PairRDD.saveAsNewHadoopDataset function to save data to HBase
<p>I want to save a Twitter stream in a HBase database. What I have now, is the Saprk Application to receive and transform the data. But I don't know how to save my TwitterStream into HBase?</p>

<p>The only thing I found that could be useful is the <code>PairRDD.saveAsNewAPIHadoopDataset(conf)</code> method. But how shall I use it, which Configurations do I have to make to able to save the RDD data to my HBase table?</p>

<p>The only thing I found yet is the HBase client library, which can insert data to a table via Put objects. But this isn't a solution for inside a Spark program, is it (would be necessary to iterate over all items inside the RDD!!)? </p>

<p>Can someone give an example in JAVA? My main problem seems to be the set-up of the <code>org.apache.hadoop.conf.Configuration</code> instance, I have to submit in the <code>saveAsNewAPIHadoopDataset</code>...</p>

<p>Here a code snippet:</p>

<pre><code>JavaReceiverInputDStream&lt;Status&gt; statusDStream = TwitterUtils.createStream(streamingCtx);

JavaPairDStream&lt;Long, String&gt; statusPairDStream = statusDStream.mapToPair(new PairFunction&lt;Status, Long, String&gt;() {
    public Tuple2&lt;Long, String&gt; call(Status status) throws Exception {
        return new Tuple2&lt;Long, String&gt; (status.getId(), status.getText());
    }
});

statusPairDStream.foreachRDD(new Function&lt;JavaPairRDD&lt;Long,String&gt;, Void&gt;() {
    public Void call(JavaPairRDD&lt;Long, String&gt; status) throws Exception {
        org.apache.hadoop.conf.Configuration conf = new Configuration();
        status.saveAsNewAPIHadoopDataset(conf);
        // HBase PUT here can't be correct!?
        return null;
    }
});
</code></pre>