Writing a sequence file from an image in local to HDFS using Java and Spark
<p>As the title says, that is my objective now. </p>

<ul>
<li>I need to load a bunch of non-text files from a directory</li>
<li>extract the usual file information from them (creation date, author, type... those ones)</li>
<li>Create a sequence file of the type </li>
<li>put the fresh extracted info in the Key of the .seq file</li>
<li>store all of them in a hdfs directory.</li>
</ul>

<p>The reason I'm using spark is for scalability (thousands of files to process and I will have a cluster of worker available) and because I'm thinking of implementing a SParkStreaming receiver on the image directory, so that the files will be processed automatically. 
Here is my initial code:</p>

<pre><code>JavaPairRDD&lt;String, String&gt; imageRDD = jsc.wholeTextFiles("file:///home/cloudera/Pictures/");

    imageRDD.mapToPair(new PairFunction&lt;Tuple2&lt;String,String&gt;, Text, Text&gt;() {

        @Override
        public Tuple2&lt;Text, Text&gt; call(Tuple2&lt;String, String&gt; arg0)
                throws Exception {
            return new Tuple2&lt;Text, Text&gt;(new Text(arg0._1),new Text(arg0._2));
        }

    }).saveAsNewAPIHadoopFile("hdfs://localhost:8020/user/hdfs/sparkling/try.seq", Text.class, Text.class, SequenceFileOutputFormat.class);
</code></pre>

<p>Here I load an image as a text file and create a tuple with the Text type from the hadoop library. This works, but:  </p>

<ol>
<li>The file isn't saved as a single one, but as a folder containing the partitions. </li>
<li>It isn't an array of byte, but a text representation of the file. We all know how nagging it can be to reconvert from text to image (or whatever it is)</li>
<li>If I load the files like this, Will there be a way to extract the required information?</li>
</ol>

<p>I've tried to load the files as aa<code>sparkContext.binaryFiles(&lt;directory&gt;)</code>, but I'm always lost as how to extract info and on how to save them.<br>
I can't seem to find the answer in the internet: does anybody of you know something about this?</p>