SparkException Task Not Serializable With Broadcast Variable (Version 1.5.2)
<p>I have</p>

<pre><code>scala&gt; sks
res32: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[10] at filter at &lt;console&gt;:45

scala&gt; sks2
res33: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[15] at map at &lt;console&gt;:45
</code></pre>

<p><code>
    scala&gt; sks.collect
    res22: Array[String] = Array(javascript, java, python, c#, c++, sql, linux, css, html, angularjs, ios, mysql, amazon-web-services, php, ruby, ruby-on-rails, node.js, android, .net, jquery, sysadmin, objective-c, postgresql, html5, c, nosql, rest, chef, hadoop, asp.net, reactjs, agile, scala, devops, mongodb, go, git, django, unix, swift, elasticsearch, user-interface, api, sql-server, puppet, css3, backbone.js, spring, asp.net-mvc, mobile, automation, json, bigdata, tdd, jenkins, cloud, selenium, apache, perl, azure, testing, apache-spark, machine-learning, qa, bash, sass, web-services, redis, docker, security, rdbms, r, jira, database, oracle, xml, tomcat, mvc, saas, open-source, twitter-bootstrap, multithreading, architecture, java-ee, aws, soap, project-management, soa, ajax, wpf, mo...
</code></p>

<pre><code>scala&gt; sks2.collect
res20: Array[String] = Array(.net, .net-4.5, 3d, 3d-modelling, 3d-printing, C/C++, ability-to-code-in-c#-4.0, accelos, access, actionscript, active-directory, adobe, adobe-illustrator, agile, agile-processes, agile-project-management, agile-scrum, ajax, akeneo, akka, algorithm, algorithmic-trading, algorithms, amazon, amazon-cloudfront, amazon-ec2, amazon-elb, amazon-redshift, amazon-s3, amazon-web-services, analysis, analytics, android, android-activity, android-fragments, android-layout, android-sdk-tools, android-volley, angular, angular-fullstack, angular-ui-bootstrap, angular.js, angularjs, ansible, ant, apache, apache-kafka, apache-spark, apache-spark-sql, apache-storm, apex, api, api-design, appium, apple-push-notifications, apple-watch, application, ar, architechture, architectu...
scala&gt;
</code></pre>

<p>`</p>

<p>I set:</p>

<p><code>
    val bcsk = sc.broadcast(sks.collect)
    bcsk: org.apache.spark.broadcast.Broadcast[Array[String]] = Broadcast(14)
</code></p>

<p>Then I paste this following in the REPL:</p>

<p><code>
    scala&gt; val docs = wtf.values.
         |   map(text =&gt; { 
         |     var i = 0
         |     var j = 0
         |     var cc = 0
         |     var cnt = Array[String]()
         |     var copyText = text.toLowerCase
         |     for(i &lt;- 0 until bcsk.value.size) {
         |       var searchText = bcsk.value(i) 
         |       if((searchText == "apache-spark") ||
         |         (searchText == "apache-kafka") ||
         |         (searchText == "apache-storm")) {
         |         searchText = searchText.replaceAll("\\Qapache-\\E", "")
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |       }
         | 
         |       if(searchText.contains('-')) {
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |         printf("has -: cc = %d, skill = %s, cnt = %d\n", cc, searchText, cnt.size)
         |         if(cc &gt; 0) {
         |           copyText = copyText.replaceAll("\\Q" + bcsk.value(i) + "\\E", "")
         |         }
         |         // now look for same text without '-'
         |         searchText = searchText.replace('-', ' ')
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |         if(cc &gt; 0) {
         |           copyText = copyText.replaceAll( "\\Q" + bcsk.value(i) + "\\E", "")
         |         }
         |       } // if search text with a '-'
         | 
         |       if((searchText == "c++") ||
         |         (searchText == "c#") ||
         |         (searchText == "f#")) {
         |         var searchTextCopy = "\\Q"+searchText + "\\E"
         |         cc = searchTextCopy.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |       }<br>
         |     } // for
         |     var words = copyText.split("\\W+")
         |     for(i &lt;- 0 until bcsk.value.size) {
         |       for(j &lt;- 0 until words.size) {
         |           if(words(j) == bcsk.value(i)) {
         |               cnt = cnt :+ bcsk.value(i)
         |           }
         |       }
         |     }
         |     cnt<br>
         |   }).
         |   map(x =&gt; x.toSeq)
    docs: org.apache.spark.rdd.RDD[Seq[String]] = MapPartitionsRDD[27] at map at &lt;console&gt;:110
</code></p>

<p>Which works (no stack trace), but is not beautiful code ... but is what it is ...</p>

<p>However if I change this to:
<code>
    scala&gt; val bcsk = sc.broadcast(sks2.collect)
    bcsk: org.apache.spark.broadcast.Broadcast[Array[String]] = Broadcast(34)
</code></p>

<p>and when I paste the same aforementioned code into the REPL, I get:</p>

<p><code>
    scala&gt; val docs = wtf.values.
         |   map(text =&gt; { 
         |     var i = 0
         |     var j = 0
         |     var cc = 0
         |     var cnt = Array[String]()
         |     var copyText = text.toLowerCase
         |     for(i &lt;- 0 until bcsk.value.size) {
         |       var searchText = bcsk.value(i) 
         |       if((searchText == "apache-spark") ||
         |         (searchText == "apache-kafka") ||
         |         (searchText == "apache-storm")) {
         |         searchText = searchText.replaceAll("\\Qapache-\\E", "")
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |       }
         | 
         |       if(searchText.contains('-')) {
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |         printf("has -: cc = %d, skill = %s, cnt = %d\n", cc, searchText, cnt.size)
         |         if(cc &gt; 0) {
         |           copyText = copyText.replaceAll("\\Q" + bcsk.value(i) + "\\E", "")
         |         }
         |         // now look for same text without '-'
         |         searchText = searchText.replace('-', ' ')
         |         cc = searchText.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |         if(cc &gt; 0) {
         |           copyText = copyText.replaceAll( "\\Q" + bcsk.value(i) + "\\E", "")
         |         }
         |       } // if search text with a '-'
         | 
         |       if((searchText == "c++") ||
         |         (searchText == "c#") ||
         |         (searchText == "f#")) {
         |         var searchTextCopy = "\\Q"+searchText + "\\E"
         |         cc = searchTextCopy.r.findAllIn(copyText).length
         |         for(j &lt;- 0 until cc) {
         |           cnt = cnt :+ bcsk.value(i) 
         |         }
         |       }<br>
         |     } // for
         |     var words = copyText.split("\\W+")
         |     for(i &lt;- 0 until bcsk.value.size) {
         |       for(j &lt;- 0 until words.size) {
         |           if(words(j) == bcsk.value(i)) {
         |               cnt = cnt :+ bcsk.value(i)
         |           }
         |       }
         |     }
         |     cnt<br>
         |   }).
         |   map(x =&gt; x.toSeq)
</code></p>

<p>with Stack trace:
<code>
    org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2032)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:318)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:317)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
        at org.apache.spark.rdd.RDD.map(RDD.scala:317)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:54)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:117)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:119)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:121)
        at $iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:123)
        at $iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:125)
        at $iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:127)
        at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:129)
        at $iwC.&lt;init&gt;(&lt;console&gt;:131)
        at &lt;init&gt;(&lt;console&gt;:133)
        at .&lt;init&gt;(&lt;console&gt;:137)
        at .&lt;clinit&gt;(&lt;console&gt;)
        at .&lt;init&gt;(&lt;console&gt;:7)
        at .&lt;clinit&gt;(&lt;console&gt;)
        at $print(&lt;console&gt;)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:875)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.io.NotSerializableException: org.apache.hadoop.conf.Configuration
    Serialization stack:
        - object not serializable (class: org.apache.hadoop.conf.Configuration, value: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, hbase-default.xml, hbase-site.xml, file:/home/ec2-user/conf/hbase-site.xml)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: conf, type: class org.apache.hadoop.conf.Configuration)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC@26ce742b)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC@23a5078f)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC@36b819)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC@188e789a)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC@f766aea)
        - field (class: $iwC$$iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC@1b0c819)
        - field (class: $iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC, $iwC$$iwC$$iwC@2105c149)
        - field (class: $iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC)
        - object (class $iwC$$iwC, $iwC$$iwC@662c5207)
        - field (class: $iwC, name: $iw, type: class $iwC$$iwC)
        - object (class $iwC, $iwC@2cd9b65d)
        - field (class: $line44.$read, name: $iw, type: class $iwC)
        - object (class $line44.$read, $line44.$read@67f0d43a)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $VAL299, type: class $line44.$read)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC@364c2c3)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: $outer, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC@70643d4c)
        - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1, name: $outer, type: class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC)
        - object (class $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1, &lt;function1&gt;)
        at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
        ... 174 more
</code></p>

<p>Why does this happen? How can I fix this? </p>