{"action":"created","issue":{"url":"https://api.github.com/repos/damballa/abracad/issues/15","labels_url":"https://api.github.com/repos/damballa/abracad/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/damballa/abracad/issues/15/comments","events_url":"https://api.github.com/repos/damballa/abracad/issues/15/events","html_url":"https://github.com/damballa/abracad/pull/15","id":65896587,"number":15,"title":"This fixes a problem when used with Apache Spark / AOT compiled code.","user":{"login":"chrisbetz","id":1052643,"avatar_url":"https://avatars.githubusercontent.com/u/1052643?v=3","gravatar_id":"","url":"https://api.github.com/users/chrisbetz","html_url":"https://github.com/chrisbetz","followers_url":"https://api.github.com/users/chrisbetz/followers","following_url":"https://api.github.com/users/chrisbetz/following{/other_user}","gists_url":"https://api.github.com/users/chrisbetz/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisbetz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisbetz/subscriptions","organizations_url":"https://api.github.com/users/chrisbetz/orgs","repos_url":"https://api.github.com/users/chrisbetz/repos","events_url":"https://api.github.com/users/chrisbetz/events{/privacy}","received_events_url":"https://api.github.com/users/chrisbetz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-02T10:07:26Z","updated_at":"2015-04-02T10:40:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/damballa/abracad/pulls/15","html_url":"https://github.com/damballa/abracad/pull/15","diff_url":"https://github.com/damballa/abracad/pull/15.diff","patch_url":"https://github.com/damballa/abracad/pull/15.patch"},"body":"Hi!\r\n\r\nI had a problem using custom avro-readers in Spark, as the record_reader attempted to call an unbound function (See stack trace below).\r\n\r\nRequiring the namespace of the var (line 26) fixed the problem.\r\n\r\nApparently, it has something to do with AOT-compilation, though I wasn't able to reproduce the error in a test case with proper effort. :(\r\n\r\nI also hit format code, so sorry, some whitespaces are different. It's only the new line 26 I added.\r\n\r\nSincerly,\r\n\r\nChris\r\n\r\n\r\n\r\n2015-04-02 11:56:00.314 [Result resolver thread-1                ] WARN  org.apache.spark.scheduler.TaskSetManager                   : Lost task 0.1 in stage 1.0 (TID 5, trillian.pmd.local): java.lang.IllegalStateException: Attempting to call unbound fn: #'pmd.pacs.data.impression.rdd/performance-data-item-from-impressions\r\n        clojure.lang.Var$Unbound.throwArity(Var.java:43)\r\n        clojure.lang.AFn.invoke(AFn.java:62)\r\n        clojure.lang.Var.invoke(Var.java:430)\r\n        clojure.lang.AFn.applyToHelper(AFn.java:195)\r\n        clojure.lang.Var.applyTo(Var.java:700)\r\n        clojure.core$apply.invoke(core.clj:628)\r\n        abracad.avro.read$record_reader$fn__302.invoke(read.clj:24)\r\n        abracad.avro.read$read_record.invoke(read.clj:53)\r\n        clojure.lang.Var.invoke(Var.java:388)\r\n        abracad.avro.ClojureDatumReader.readRecord(ClojureDatumReader.java:57)\r\n        org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)\r\n        abracad.avro.ClojureDatumReader.read(ClojureDatumReader.java:50)\r\n        org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)\r\n        org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)\r\n        org.apache.avro.mapreduce.AvroRecordReaderBase.nextKeyValue(AvroRecordReaderBase.java:118)\r\n        sparkling.hadoop.ClojureAvroRecordReader.nextKeyValue(ClojureAvroRecordReader.java:27)"},"comment":{"url":"https://api.github.com/repos/damballa/abracad/issues/comments/88863129","html_url":"https://github.com/damballa/abracad/pull/15#issuecomment-88863129","issue_url":"https://api.github.com/repos/damballa/abracad/issues/15","id":88863129,"user":{"login":"llasram","id":185528,"avatar_url":"https://avatars.githubusercontent.com/u/185528?v=3","gravatar_id":"","url":"https://api.github.com/users/llasram","html_url":"https://github.com/llasram","followers_url":"https://api.github.com/users/llasram/followers","following_url":"https://api.github.com/users/llasram/following{/other_user}","gists_url":"https://api.github.com/users/llasram/gists{/gist_id}","starred_url":"https://api.github.com/users/llasram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llasram/subscriptions","organizations_url":"https://api.github.com/users/llasram/orgs","repos_url":"https://api.github.com/users/llasram/repos","events_url":"https://api.github.com/users/llasram/events{/privacy}","received_events_url":"https://api.github.com/users/llasram/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T10:40:40Z","updated_at":"2015-04-02T10:40:40Z","body":"I'm open to alternatives, but at present this is by design. Performing the `require` as part of every record de-serialization does not seem like the right solution, especially given that the set of reader functions should rarely/never change at runtime. There might be a clean way to do it e.g. as part of construction of the `DatumReader`, but thus far I'm following the design of the Clojure core tagged literal readers feature, which also leaves loading necessary namespaces to the user.\r\n\r\nIn Parkour for Hadoop MapReduce I've solved this problem by having the local-side Clojure code set a configuration parameter listing all locally-loaded namespaces, then having the cluster-side task driver attempt to `require` all of those namespaces. You hopefully should be able to implement a similar solution for Spark."}}