org.apache.spark.SparkException: Task not serializable in Spark Scala
<p>I am trying to get <code>employeeId</code> from <code>employee_table</code> and use this id to query <code>employee_address</code> table to fetch the address.</p>
<p>There is nothing wrong with tables. But when I run the below code, I get <code>org.apache.spark.SparkException: Task not serializable</code></p>
<p>I think I know the issue. The issue is sparkContext is with master and not with worker. But I don't know how to get my head around this.</p>
<pre><code>val employeeRDDRdd = sc.cassandraTable(&quot;local_keyspace&quot;, &quot;employee_table&quot;)


try {

  val data = employeeRDDRdd
    .map(row =&gt; {
      row.getStringOption(&quot;employeeID&quot;) match {
        case Some(s) if (s != null) &amp;&amp; s.nonEmpty =&gt; s
        case None =&gt; &quot;&quot;
      }
    })

    //create tuple of employee id and address. Filtering out cases when  for an employee address is empty.

  val id = data
    .map(s =&gt; (s,getID(s)))
    filter(tups =&gt; tups._2.nonEmpty)

    //printing out total size of rdd.
    println(id.count())




} catch {
  case e: Exception =&gt; e.printStackTrace()
}

def getID(employeeID: String): String = {
  val addressRDD = sc.cassandraTable(&quot;local_keyspace&quot;, &quot;employee_address&quot;)
  val data = addressRDD.map(row =&gt; row.getStringOption(&quot;address&quot;) match {
    case Some(s) if (s != null) &amp;&amp; s.nonEmpty =&gt; s
    case None =&gt; &quot;&quot;
  })
  data.collect()(0)
}
</code></pre>
<p><strong>Exception</strong> ==&gt;</p>
<pre><code>rg.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2039)
    at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:366)
    at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:365)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    at org.apache.spark.rdd.RDD.map(RDD.scala:365)
</code></pre>