How to serialize the data to AVRO schema in Spark (with Java)?
<p>I have defined an AVRO schema, and generated some classes with avro-tools for the schemes. Now, I want to serialize the data to disk. I found some answers about scala for this, but not for Java. The class <code>Article</code> is generated with avro-tools, and is made from a schema defined by me.</p>

<p>Here's a simplified version of the code of how I try to do it:</p>

<pre><code>JavaPairRDD&lt;String, String&gt; filesRDD = context.wholeTextFiles(inputDataPath);
JavaRDD&lt;Article&gt; processingFiles = filesRDD.map(fileNameContent -&gt; {
    // The name of the file
    String fileName = fileNameContent._1();
    // The content of the file
    String fileContent = fileNameContent._2();

    // An object from my avro schema
    Article a = new Article(fileContent);

    Processing processing = new Processing();
    // .... some processing of the content here ... //

    processing.serializeArticleToDisk(avroFileName);

    return a;
});
</code></pre>

<p>where <code>serializeArticleToDisk(avroFileName)</code> is defined as follows:</p>

<pre><code>public void serializeArticleToDisk(String filename) throws IOException{
    // Serialize article to disk
    DatumWriter&lt;Article&gt; articleDatumWriter = new SpecificDatumWriter&lt;Article&gt;(Article.class);
    DataFileWriter&lt;Article&gt; dataFileWriter = new DataFileWriter&lt;Article&gt;(articleDatumWriter);
    dataFileWriter.create(this.article.getSchema(), new File(filename));
    dataFileWriter.append(this.article);
    dataFileWriter.close();
}
</code></pre>

<p>where <code>Article</code> is my avro schema.</p>

<p>Now, the mapper throws me the error:</p>

<pre><code>java.io.FileNotFoundException: hdfs:/...path.../avroFileName.avro (No such file or directory)   
at java.io.FileOutputStream.open0(Native Method)    
at java.io.FileOutputStream.open(FileOutputStream.java:270)     
at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213)   
at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:162)   
at org.apache.avro.file.SyncableFileOutputStream.&lt;init&gt;(SyncableFileOutputStream.java:60)   
at org.apache.avro.file.DataFileWriter.create(DataFileWriter.java:129)
at org.apache.avro.file.DataFileWriter.create(DataFileWriter.java:129)
at sentences.ProcessXML.serializeArticleToDisk(ProcessXML.java:207)     
. . . rest of the stacktrace ... 
</code></pre>

<p>although the file path is correct. </p>

<p>I use a <code>collect()</code> method afterwards, so everything else within the <code>map</code> function works fine (except for the serialization part). </p>

<p>I am quite new with Spark, so I am not sure if this might be something trivial actually. I suspect that I need to use some writing functions, not to do the writing in the mapper (not sure if this is true, though). Any ideas how to tackle this?</p>

<p><strong>EDIT:</strong></p>

<p>The last line of the error stack-trace I showed, is actually on this part:</p>

<p><code>dataFileWriter.create(this.article.getSchema(), new File(filename));</code></p>

<p>This is the part that throws the actual error. I am assuming the <code>dataFileWriter</code> needs to be replaced with something else. Any ideas?</p>