Converting a Spark's DataFrame column to List[String] in Scala
<p>I am working on <a href="http://grouplens.org/datasets/movielens/" rel="nofollow" title="Movie Lens">Movie Lens</a> data set. In one the the <code>csv</code> files, the data is structured as:</p>

<p><code>movieId</code> <code>movieTitle</code> <code>genres</code></p>

<p>and <code>genres</code> again is a list of <code>|</code> separated values, the field is nullable.</p>

<p>I am trying to get a unique list of all the <code>genres</code> so that I can rearrange the data as following:</p>

<p><code>movieId</code> <code>movieTitle</code> <code>genre1</code> <code>genre2</code> <code>...</code> <code>genreN</code></p>

<p>and a row, which has <code>genre</code> as <code>genre1 | genre2</code> will look like:</p>

<p><code>1</code> <code>Title1</code> <code>1</code> <code>1</code> <code>0</code> <code>...</code> <code>0</code> </p>

<p>So far, I have been able to read the <code>csv</code> file using the following code:</p>

<pre><code>val conf         = new SparkConf().setAppName(App.name).setMaster(App.sparkMaster)
val context      = new SparkContext(conf)
val sparkSession = SparkSession.builder()
                   .appName(App.name)
                   .config("header", "true")
                   .config(conf = conf)
                   .getOrCreate()

val movieFrame: DataFrame = sparkSession.read.csv(moviesPath)
</code></pre>

<p>If I try something like:</p>

<pre><code>movieFrame.rdd.map(row ⇒ row(2).asInstanceOf[String]).collect()
</code></pre>

<p>Then I get the following exception:</p>

<pre><code>java.lang.ClassNotFoundException: com.github.babbupandey.ReadData$$anonfun$1
</code></pre>

<p>Then, in addition, I tried providing the schema explicitly using the following code:</p>

<pre><code>val moviesSchema: StructType = StructType(Array(StructField("movieId", StringType, nullable = true),
                                                        StructField("title", StringType, nullable = true),
                                                        StructField("genres", StringType, nullable = true)))
</code></pre>

<p>and tried:</p>

<pre><code>val movieFrame: DataFrame = sparkSession.read.schema(moviesSchema).csv(moviesPath)
</code></pre>

<p>and then I got the same exception.</p>

<p>Is there any way in which I can the set of <code>genres</code> as a <code>List</code> or a <code>Set</code> so I can further massage the data into the desired format? Any help will be appreciated.</p>