Apache Spark: Read umlauts with SparkContext.textFile
<p>I wrote a Spark Application which reads, filters and prints data from a CSV file, saved in HDFS. When I download the CSV to my pc and open it, everything is ok, all umlauts are displayed correctly. BUT: When I run my application, the umlauts are not shown correctly, they all are displayed as <code>ï¿½</code>.</p>

<p>Is there a possibility to set an encoding for loading the csv via .textFile transaction in Spark or to "convert" the input?</p>

<p><strong>This seems to be a Won't Fix:</strong> <a href="https://issues.apache.org/jira/browse/SPARK-1849" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-1849</a></p>

<p>The problem occurs due to the UTF-8 encoding of the file. Seems, that there's a workaround by using <code>.hadoopFile</code>, but I don't know how to use this method. Can someone help?</p>

<p><strong>Tried the following code, without success:</strong></p>

<p>test.txt file:</p>

<pre><code>ÄPPLE
APPLE
ÖPPLE
ÜPPLE
</code></pre>

<p>Spark program:</p>

<pre><code>public static void main(String[] args) throws ClassNotFoundException {
    String inputFile = args[0];

    SparkConf sparkConf = new SparkConf().setAppName("EdekaOwgCheckUmlauts");
    sparkConf.registerKryoClasses(new Class&lt;?&gt;[]{
        Class.forName("org.apache.hadoop.io.LongWritable"),
        Class.forName("org.apache.hadoop.io.Text")
    });

    JavaSparkContext sc = new JavaSparkContext(sparkConf);

    JavaRDD&lt;Text&gt; input = sc.hadoopFile(inputFile, TextInputFormat.class, LongWritable.class, Text.class).map(new Function&lt;Tuple2&lt;LongWritable,Text&gt;, Text&gt;() {
        @Override
        public Text call(Tuple2&lt;LongWritable, Text&gt; v1) throws Exception {
            return v1._2();
        }
    });

    JavaRDD&lt;Text&gt; filtered = input.filter(new Function&lt;Text, Boolean&gt;() {
        @Override
        public Boolean call(Text v1) throws Exception {
            return v1.toString().toLowerCase().contains("pple");        // LOOK FOR ÄPPLE, APPLE etc.
        }
    });

    List&lt;Text&gt; collected = filtered.collect();
    System.out.println("=== PRINT ===");
    for(Text row : collected) {
        try {
            System.out.println(row.decode(row.getBytes()));
        } catch (CharacterCodingException e) {
            e.printStackTrace();
        }
    }
}
</code></pre>

<p>Console Output:</p>

<pre><code>ï¿½PPLE
APPLE
ï¿½PPLE
ï¿½PPLE
</code></pre>

<p><strong>The file is encoded with the charset iso-8859-1</strong></p>

<p>Calling <code>file -i myfile.csv</code> shows the following information:</p>

<pre><code>myfile.csv: text/plain; charset=iso-8859-1
</code></pre>

<p>I tried to convert this charset into utf-8 using this code:</p>

<pre><code>JavaRDD&lt;String&gt; inputRDD = sc.textFile(inputFile);
List&lt;String&gt; asList = inputRDD.collect();
for(String a : asList) {

    try {
        byte[] isoBytes = a.getBytes("ISO-8859-1");
        System.out.println(new String(isoBytes, "UTF-8"));
    } catch (Exception e) {
        e.printStackTrace();
    }
}
</code></pre>

<p>without success, the console prints <code>?</code> instead ä, ö, ü...</p>