{"action":"created","issue":{"url":"https://api.github.com/repos/sryza/aas/issues/44","labels_url":"https://api.github.com/repos/sryza/aas/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/sryza/aas/issues/44/comments","events_url":"https://api.github.com/repos/sryza/aas/issues/44/events","html_url":"https://github.com/sryza/aas/issues/44","id":104945366,"number":44,"title":"Ch06 - org.apache.spark.SparkException: Task not serializable","user":{"login":"jackRogers","id":1609589,"avatar_url":"https://avatars.githubusercontent.com/u/1609589?v=3","gravatar_id":"","url":"https://api.github.com/users/jackRogers","html_url":"https://github.com/jackRogers","followers_url":"https://api.github.com/users/jackRogers/followers","following_url":"https://api.github.com/users/jackRogers/following{/other_user}","gists_url":"https://api.github.com/users/jackRogers/gists{/gist_id}","starred_url":"https://api.github.com/users/jackRogers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackRogers/subscriptions","organizations_url":"https://api.github.com/users/jackRogers/orgs","repos_url":"https://api.github.com/users/jackRogers/repos","events_url":"https://api.github.com/users/jackRogers/events{/privacy}","received_events_url":"https://api.github.com/users/jackRogers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-09-04T18:39:04Z","updated_at":"2015-09-14T18:31:04Z","closed_at":null,"body":"The book example uses the path to wikidump.xml, but the github code is looking at a directory.  Where and how was the xml file broken up?  I'm getting this error in the preprocessing function when running flatmap. \r\n\r\norg.apache.spark.SparkException: Task not serializable\r\n\tat org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:315)\r\n\tat org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:305)\r\n\tat org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:132)\r\n\tat org.apache.spark.SparkContext.clean(SparkContext.scala:1891)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$flatMap$1.apply(RDD.scala:303)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$flatMap$1.apply(RDD.scala:302)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109)\r\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:286)\r\n\tat org.apache.spark.rdd.RDD.flatMap(RDD.scala:302)\r\n\tat $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$RunLSA$.preprocessing(<console>:234)\r\n\tat $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$RunLSA$.main(<console>:65)\r\n\r\n\r\nAdditionally, is there any documentation on how to run RunLSA? The book example uses spark-shell but I've had to change a few things to get the github code to play nicely with spark-shell"},"comment":{"url":"https://api.github.com/repos/sryza/aas/issues/comments/140167941","html_url":"https://github.com/sryza/aas/issues/44#issuecomment-140167941","issue_url":"https://api.github.com/repos/sryza/aas/issues/44","id":140167941,"user":{"login":"srowen","id":822522,"avatar_url":"https://avatars.githubusercontent.com/u/822522?v=3","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"created_at":"2015-09-14T18:31:04Z","updated_at":"2015-09-14T18:31:04Z","body":"I didn't say that; the snippets in the book are meant to be used, as snippets, which might help write your own similar code. The Github repo is supposed to be the same code but with all the extra details around it filled in so it can compile normally, and some extra structure. It's also meant to be used in the same sense. I'm not sure running the code from a chapter end to end does something useful other than execute all of the illustrations in order, yes. None of the code is a library. You'll want to lift and adapt sections to create your own application. If there's a problem in your app that you're pretty sure manifests in the code here too, we can have a look; if it's really your app code, might be out of scope."}}