{"action":"created","issue":{"url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/134","labels_url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/134/comments","events_url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/134/events","html_url":"https://github.com/ibm-et/spark-kernel/issues/134","id":106401203,"number":134,"title":"sc.addJar does not add JAR","user":{"login":"pdendek","id":2032482,"avatar_url":"https://avatars.githubusercontent.com/u/2032482?v=3","gravatar_id":"","url":"https://api.github.com/users/pdendek","html_url":"https://github.com/pdendek","followers_url":"https://api.github.com/users/pdendek/followers","following_url":"https://api.github.com/users/pdendek/following{/other_user}","gists_url":"https://api.github.com/users/pdendek/gists{/gist_id}","starred_url":"https://api.github.com/users/pdendek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdendek/subscriptions","organizations_url":"https://api.github.com/users/pdendek/orgs","repos_url":"https://api.github.com/users/pdendek/repos","events_url":"https://api.github.com/users/pdendek/events{/privacy}","received_events_url":"https://api.github.com/users/pdendek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ibm-et/spark-kernel/labels/investigate","name":"investigate","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-14T18:32:03Z","updated_at":"2015-09-15T09:11:54Z","closed_at":null,"body":"A quick test:\r\nsc.addJar(\"/home/joe/myjar.jar\")\r\nsc.jars\r\n>> List()\r\n\r\nThis issue has been observed with Spark 1.5.0 on Ubuntu 14.04. with spark-kernel\r\nAdding JAR files from a notebook is a crucial feature in many applications\r\n(esp. of my own)\r\n\r\nLooking forward for the feedback."},"comment":{"url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/comments/140332451","html_url":"https://github.com/ibm-et/spark-kernel/issues/134#issuecomment-140332451","issue_url":"https://api.github.com/repos/ibm-et/spark-kernel/issues/134","id":140332451,"user":{"login":"pdendek","id":2032482,"avatar_url":"https://avatars.githubusercontent.com/u/2032482?v=3","gravatar_id":"","url":"https://api.github.com/users/pdendek","html_url":"https://github.com/pdendek","followers_url":"https://api.github.com/users/pdendek/followers","following_url":"https://api.github.com/users/pdendek/following{/other_user}","gists_url":"https://api.github.com/users/pdendek/gists{/gist_id}","starred_url":"https://api.github.com/users/pdendek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdendek/subscriptions","organizations_url":"https://api.github.com/users/pdendek/orgs","repos_url":"https://api.github.com/users/pdendek/repos","events_url":"https://api.github.com/users/pdendek/events{/privacy}","received_events_url":"https://api.github.com/users/pdendek/received_events","type":"User","site_admin":false},"created_at":"2015-09-15T09:11:54Z","updated_at":"2015-09-15T09:11:54Z","body":"Yes, I've tested if the file has been added, but apparently it is not loaded\r\n> Name: Compile Error\r\n> Message: <console>:14: error: not found: value pl\r\n>       import my.package._\r\n>              ^\r\n> StackTrace: \r\n\r\n\r\nWhen I tried to sort adding JARs out in a different way\r\nI took the approach to stop the \"main\" context (sc)\r\nand create the new one (sCtx; the one that would allow me to add JAR to the config object\r\nand eventually to SparkContext). However I get errors which doesn't occur \r\nwhen using the \"main\" context (sc).\r\nThe code is as follows:\r\n```scala\r\nimport org.apache.spark.SparkContext\r\nimport org.apache.spark.SparkContext._\r\nimport org.apache.spark.SparkConf\r\nsc.stop\r\nval conf = new SparkConf(true)\r\nval sCtx = new SparkContext(\"local[4]\", \"test\", conf)\r\n\r\nval BASE_DIR = \"/home/pdendek/\"\r\nval rawIris = (sCtx\r\n               .textFile(BASE_DIR + \"sample_data/iris.csv\")\r\n               .filter(! _.trim.isEmpty)\r\n               .map(_.split(\",\"))\r\n               .map(p => (p(4), p.take(4).map(_.toDouble))))\r\nimport org.apache.spark.mllib.linalg.Vectors\r\nimport org.apache.spark.mllib.regression.LabeledPoint\r\nval names = rawIris.map(_._1).distinct.zipWithIndex.collectAsMap\r\nval bcNames = sc.broadcast(names)\r\nval lp = rawIris.map(r => LabeledPoint(bcNames.value(r._1),Vectors.dense(r._2)))\r\n```\r\n\r\nWith the StackTrace as follows:\r\n\r\n```\r\nbcNames: org.apache.spark.broadcast.Broadcast[scala.collection.Map[String,Long]] = Broadcast(4)\r\norg.apache.spark.SparkException: Task not serializable\r\n        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)\r\n        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)\r\n        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)\r\n        at org.apache.spark.SparkContext.clean(SparkContext.scala:2021)\r\n        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:314)\r\n        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:313)\r\n        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)\r\n        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)\r\n        at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)\r\n        at org.apache.spark.rdd.RDD.map(RDD.scala:313)\r\n        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:26)\r\n        at $iwC$$iwC$$iwC.<init>(<console>:31)\r\n        at $iwC$$iwC.<init>(<console>:33)\r\n        at $iwC.<init>(<console>:35)\r\n        at <init>(<console>:37)\r\n        at .<init>(<console>:41)\r\n        at .<clinit>(<console>)\r\n        at .<init>(<console>:7)\r\n        at .<clinit>(<console>)\r\n        at $print(<console>)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:497)\r\n        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)\r\n        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)\r\n        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)\r\n        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)\r\n        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)\r\n        at com.ibm.spark.interpreter.ScalaInterpreter$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreter.scala:296)\r\n        at com.ibm.spark.interpreter.ScalaInterpreter$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreter.scala:291)\r\n        at com.ibm.spark.global.StreamState$.withStreams(StreamState.scala:80)\r\n        at com.ibm.spark.interpreter.ScalaInterpreter$$anonfun$interpretAddTask$1.apply(ScalaInterpreter.scala:290)\r\n        at com.ibm.spark.interpreter.ScalaInterpreter$$anonfun$interpretAddTask$1.apply(ScalaInterpreter.scala:290)\r\n        at com.ibm.spark.utils.TaskManager$$anonfun$add$2$$anon$1.run(TaskManager.scala:123)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.NotSerializableException: org.apache.spark.SparkContext\r\nSerialization stack:\r\n        - object not serializable (class: org.apache.spark.SparkContext, value: org.apache.spark.SparkContext@18bb1b88)\r\n        - field (class: $iwC$$iwC$$iwC$$iwC, name: mSc, type: class org.apache.spark.SparkContext)\r\n        - object (class $iwC$$iwC$$iwC$$iwC, $iwC$$iwC$$iwC$$iwC@79e54d90)\r\n        - field (class: $iwC$$iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC$$iwC)\r\n        - object (class $iwC$$iwC$$iwC, $iwC$$iwC$$iwC@7da382c2)\r\n        - field (class: $iwC$$iwC, name: $iw, type: class $iwC$$iwC$$iwC)\r\n        - object (class $iwC$$iwC, $iwC$$iwC@4b22a10e)\r\n        - field (class: $iwC, name: $iw, type: class $iwC$$iwC)\r\n        - object (class $iwC, $iwC@6cf94088)\r\n        - field (class: $line10.$read, name: $iw, type: class $iwC)\r\n        - object (class $line10.$read, $line10.$read@3df83828)\r\n        - field (class: $iwC$$iwC$$iwC, name: $VAL24, type: class $line10.$read)\r\n        - object (class $iwC$$iwC$$iwC, $iwC$$iwC$$iwC@3ba027c)\r\n```\r\nIs it entirely impossible to create a new SparkContext in a notebook using spark-kernel?\r\nI hope it is solvable."}}