How to remove null data from JavaPairRDD
<p>I am getting the Hbase data and trying to do a spark job on it. My table has around 70k rows and each row has a column '<strong>type</strong>', which can have the values:<strong>post,comment or reply</strong>. Based on the type, I want to take out different Pair RDDs like shown below(for post).</p>

<pre><code>JavaPairRDD&lt;ImmutableBytesWritable, FlumePost&gt; postPairRDD = hBaseRDD.mapToPair(
            new PairFunction&lt;Tuple2&lt;ImmutableBytesWritable, Result&gt;, ImmutableBytesWritable, FlumePost&gt;() {
                private static final long serialVersionUID = 1L;

                public Tuple2&lt;ImmutableBytesWritable, FlumePost&gt; call(Tuple2&lt;ImmutableBytesWritable, Result&gt; arg0)
                        throws Exception {
                    FlumePost flumePost = new FlumePost();
                    ImmutableBytesWritable key = arg0._1;
                    Result result = arg0._2;
                    String type = Bytes.toString(result.getValue(Bytes.toBytes("cf"), Bytes.toBytes("t")));
                    if (type.equals("post")) {
                        return new Tuple2&lt;ImmutableBytesWritable, FlumePost&gt;(key, flumePost);
                    } else {
                        return null;
                    }

                }

            }).distinct();
</code></pre>

<p>Problem here is, For all the rows with type other than <strong>post</strong> I have to send null value which is undesirable. And iteration goes on for 70k times for all the three types, wasting the cycles. So my first question is:</p>

<p>1) What is the efficient way to do this?</p>

<p>So now after getting 70k results I put the <code>distinct()</code> method to remove the duplication of null values. So I end up having one null value object in it. I expect 20327 results but I get 20328. </p>

<p>2) Is there a way to remove this null entry from the pair RDD?</p>