{"action":"created","issue":{"url":"https://api.github.com/repos/yieldbot/flambo/issues/78","labels_url":"https://api.github.com/repos/yieldbot/flambo/issues/78/labels{/name}","comments_url":"https://api.github.com/repos/yieldbot/flambo/issues/78/comments","events_url":"https://api.github.com/repos/yieldbot/flambo/issues/78/events","html_url":"https://github.com/yieldbot/flambo/issues/78","id":97243093,"number":78,"title":"Flambo Stand-Alone Applications Broken","user":{"login":"senor-hadoop","id":7775472,"avatar_url":"https://avatars.githubusercontent.com/u/7775472?v=3","gravatar_id":"","url":"https://api.github.com/users/senor-hadoop","html_url":"https://github.com/senor-hadoop","followers_url":"https://api.github.com/users/senor-hadoop/followers","following_url":"https://api.github.com/users/senor-hadoop/following{/other_user}","gists_url":"https://api.github.com/users/senor-hadoop/gists{/gist_id}","starred_url":"https://api.github.com/users/senor-hadoop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senor-hadoop/subscriptions","organizations_url":"https://api.github.com/users/senor-hadoop/orgs","repos_url":"https://api.github.com/users/senor-hadoop/repos","events_url":"https://api.github.com/users/senor-hadoop/events{/privacy}","received_events_url":"https://api.github.com/users/senor-hadoop/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-07-25T17:55:40Z","updated_at":"2015-07-25T20:31:06Z","closed_at":null,"body":"Flambo applications run in stand-alone mode, whether in YARN(--master yarn-cluster, --master yarn-client), stand-alone cluster (--master spark://...) or locally (--master local[*]), all fail with the nearly the same stacktrace found below. Essentially, nothing with 'spark-submit' works. For those wanting to try YARN mode, an out-of-the-box YARN cluster can be found within the Hortonworks sandbox (a virtual machine): http://hortonworks.com/products/hortonworks-sandbox/\r\n\r\nSLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/snunez/filecache/10/spark-assembly-1.2.1.2.2.4.2-2-hadoop2.6.0.2.2.4.2-2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n15/07/25 03:05:38 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]\r\n15/07/25 03:05:39 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1437781501203_0001_000001\r\n15/07/25 03:05:40 INFO spark.SecurityManager: Changing view acls to: yarn,snunez\r\n15/07/25 03:05:40 INFO spark.SecurityManager: Changing modify acls to: yarn,snunez\r\n15/07/25 03:05:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, snunez); users with modify permissions: Set(yarn, snunez)\r\n15/07/25 03:05:40 INFO yarn.ApplicationMaster: Starting the user JAR in a separate Thread\r\n15/07/25 03:05:40 INFO yarn.ApplicationMaster: Waiting for spark context initialization\r\n15/07/25 03:05:40 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 0\r\nException in thread \"Driver\" java.lang.ExceptionInInitializerError\r\n\tat java.lang.Class.forName0(Native Method)\r\n\tat java.lang.Class.forName(Class.java:278)\r\n\tat clojure.lang.RT.loadClassForName(RT.java:2093)\r\n\tat clojure.lang.RT.load(RT.java:430)\r\n\tat clojure.lang.RT.load(RT.java:411)\r\n\tat clojure.core$load$fn__5066.invoke(core.clj:5641)\r\n\tat clojure.core$load.doInvoke(core.clj:5640)\r\n\tat clojure.lang.RestFn.invoke(RestFn.java:408)\r\n\tat clojure.core$load_one.invoke(core.clj:5446)\r\n\tat clojure.core$load_lib$fn__5015.invoke(core.clj:5486)\r\n\tat clojure.core$load_lib.doInvoke(core.clj:5485)\r\n\tat clojure.lang.RestFn.applyTo(RestFn.java:142)\r\n\tat clojure.core$apply.invoke(core.clj:626)\r\n\tat clojure.core$load_libs.doInvoke(core.clj:5524)\r\n\tat clojure.lang.RestFn.applyTo(RestFn.java:137)\r\n\tat clojure.core$apply.invoke(core.clj:626)\r\n\tat clojure.core$require.doInvoke(core.clj:5607)\r\n\tat clojure.lang.RestFn.invoke(RestFn.java:551)\r\n\tat flambo.api$loading__4958__auto__.invoke(api.clj:12)\r\n\tat flambo.api__init.load(Unknown Source)\r\n\tat flambo.api__init.<clinit>(Unknown Source)\r\n\tat java.lang.Class.forName0(Native Method)\r\n\tat java.lang.Class.forName(Class.java:278)\r\n\tat clojure.lang.RT.loadClassForName(RT.java:2093)\r\n\tat clojure.lang.RT.load(RT.java:430)\r\n\tat clojure.lang.RT.load(RT.java:411)\r\n\tat clojure.core$load$fn__5066.invoke(core.clj:5641)\r\n\tat clojure.core$load.doInvoke(core.clj:5640)\r\n\tat clojure.lang.RestFn.invoke(RestFn.java:408)\r\n\tat clojure.core$load_one.invoke(core.clj:5446)\r\n\tat clojure.core$load_lib$fn__5015.invoke(core.clj:5486)\r\n\tat clojure.core$load_lib.doInvoke(core.clj:5485)\r\n\tat clojure.lang.RestFn.applyTo(RestFn.java:142)\r\n\tat clojure.core$apply.invoke(core.clj:626)\r\n\tat clojure.core$load_libs.doInvoke(core.clj:5524)\r\n\tat clojure.lang.RestFn.applyTo(RestFn.java:137)\r\n\tat clojure.core$apply.invoke(core.clj:626)\r\n\tat clojure.core$require.doInvoke(core.clj:5607)\r\n\tat clojure.lang.RestFn.invoke(RestFn.java:512)\r\n\tat ict.core$loading__4958__auto__.invoke(core.clj:1)\r\n\tat ict.core__init.load(Unknown Source)\r\n\tat ict.core__init.<clinit>(Unknown Source)\r\n\tat java.lang.Class.forName0(Native Method)\r\n\tat java.lang.Class.forName(Class.java:278)\r\n\tat clojure.lang.RT.loadClassForName(RT.java:2093)\r\n\tat clojure.lang.RT.load(RT.java:430)\r\n\tat clojure.lang.RT.load(RT.java:411)\r\n\tat clojure.core$load$fn__5066.invoke(core.clj:5641)\r\n\tat clojure.core$load.doInvoke(core.clj:5640)\r\n\tat clojure.lang.RestFn.invoke(RestFn.java:408)\r\n\tat clojure.lang.Var.invoke(Var.java:379)\r\n\tat ict.core.<clinit>(Unknown Source)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:445)\r\nCaused by: java.lang.ClassNotFoundException: flambo.WriterOutputStream\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:366)\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:355)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:354)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:425)\r\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:358)\r\n\tat java.lang.Class.forName0(Native Method)\r\n\tat java.lang.Class.forName(Class.java:195)\r\n\tat flambo.utils$loading__4958__auto__.invoke(utils.clj:1)\r\n\tat flambo.utils__init.load(Unknown Source)\r\n\tat flambo.utils__init.<clinit>(Unknown Source)\r\n\t... 57 more\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/yieldbot/flambo/issues/comments/124887651","html_url":"https://github.com/yieldbot/flambo/issues/78#issuecomment-124887651","issue_url":"https://api.github.com/repos/yieldbot/flambo/issues/78","id":124887651,"user":{"login":"senor-hadoop","id":7775472,"avatar_url":"https://avatars.githubusercontent.com/u/7775472?v=3","gravatar_id":"","url":"https://api.github.com/users/senor-hadoop","html_url":"https://github.com/senor-hadoop","followers_url":"https://api.github.com/users/senor-hadoop/followers","following_url":"https://api.github.com/users/senor-hadoop/following{/other_user}","gists_url":"https://api.github.com/users/senor-hadoop/gists{/gist_id}","starred_url":"https://api.github.com/users/senor-hadoop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senor-hadoop/subscriptions","organizations_url":"https://api.github.com/users/senor-hadoop/orgs","repos_url":"https://api.github.com/users/senor-hadoop/repos","events_url":"https://api.github.com/users/senor-hadoop/events{/privacy}","received_events_url":"https://api.github.com/users/senor-hadoop/received_events","type":"User","site_admin":false},"created_at":"2015-07-25T20:31:06Z","updated_at":"2015-07-25T20:31:06Z","body":"Thanks. I've tried that, but lein gets into an infinite loop, producing an uberjar of ever increasing size. There is also a clean solution in the StackOverflow post:\n\n\n\nI find lein pom; lein jar; lein install works well when developing libraries.\n\nDo this in the library being developed and your application requiring it will use it without any :repositories foo required.\n\n________________________________\n\nAlternatively, lein do pom, jar, install is slightly more concise.\n\n________________________________\n\nThis allows calling the library like any other :dependencies [[project-name \"version\"]]\n\n\n\nCompared to the other solutions this is so simple and it works! No need to install maven. Thanks. –  dkinzer<http://stackoverflow.com/users/256854/dkinzer> Jul 28 '13 at 0:16<http://stackoverflow.com/questions/2404426/leiningen-how-to-add-dependencies-for-local-jars#comment26152207_13733562>\n\n\n\nHow do you reference the resulting jar in a lein project file? –  interstar<http://stackoverflow.com/users/8482/interstar> Sep 4 '14 at 7:46<http://stackoverflow.com/questions/2404426/leiningen-how-to-add-dependencies-for-local-jars#comment40097907_13733562>\n\n\n\nLike any other :dependencies [[project-name \"version\"]] –  roboli<http://stackoverflow.com/users/2202143/roboli> Sep 4 '14 at 17:09<http://stackoverflow.com/questions/2404426/leiningen-how-to-add-dependencies-for-local-jars#comment40117906_13733562>\n\n\n\nHowever that's not working either. Once I get things working I'll update the README.md to cover the use case of both developing Flambo and using it at the same time, which I imagine must be fairly common.\n\n\nIn the meantime though, I would love to see a working project.clj file that has this configuration. I can't be the only one doing this.\n\n\n\n\n________________________________\nFrom: Soren Macbeth <notifications@github.com>\nSent: Saturday, July 25, 2015 12:39 PM\nTo: yieldbot/flambo\nCc: Steve Nunez\nSubject: Re: [flambo] Flambo Stand-Alone Applications Broken (#78)\n\nI edited my previous comment to include info on leiningen checkouts, which\nwill do exactly what you want.\n\nOn Saturday, July 25, 2015, senor-hadoop <notifications@github.com> wrote:\n\n> The target-path isn't necessary; just something left from an experiment\n> and, as you point out, not causing any harm.\n>\n>\n> I had resource-paths in there because I want to pull in my version of\n> Flambo, rather than something from clojars (based off this StackOverflow\n> post<\n> http://stackoverflow.com/questions/2404426/leiningen-how-to-add-dependencies-for-local-jars>),\n> although on rereading that post I see there is a note that it doesn't work\n> with uberjars.\n>\n> I'd like to keep it as simple as possible, and many of the suggestions\n> I've found involve maintaining local maven repos and such.\n>\n>\n> Can anyone using a local copy of Flambo to develop projects post their\n> project.clj file? It would be a useful thing to inspect.\n>\n>\n> - Steve\n>\n>\n> ________________________________\n> From: Soren Macbeth <notifications@github.com\n> <javascript:_e(%7B%7D,'cvml','notifications@github.com');>>\n> Sent: Saturday, July 25, 2015 12:04 PM\n> To: yieldbot/flambo\n> Cc: Steve Nunez\n> Subject: Re: [flambo] Flambo Stand-Alone Applications Broken (#78)\n>\n>\n> I took a second look at your project.clj from the mailing list thread and\n> now (less tired) there a couple of oddities:\n>\n> (defproject ict \"0.1.0-SNAPSHOT\"\n> :description \"Transform ICT files to Hive\"\n> :resource-paths [\"/usr/local/src/flambo/target/flambo-0.7.0-SNAPSHOT.jar\"]\n> ; We're both developing and using flambo\n> :url \"http://hortonworks.com\"\n> :license {:name \"Eclipse Public License\"\n> :url \"http://www.eclipse.org/legal/epl-v10.html\"}\n> :dependencies [[org.clojure/clojure \"1.6.0\"]\n> [org.clojure/core.memoize \"0.5.7\"]\n> [yieldbot/serializable-fn \"0.0.9\"\n> :exclusions [com.twitter/chill-java]] ; Normally not need except when\n> compiling flambo locally\n> [clj-time \"0.10.0\"]]\n> :plugins [[cider/cider-nrepl \"0.10.0-SNAPSHOT\"]] ; Need this because of\n> cider/lein strangeness\n> :target-path \"target/%s\"\n> :profiles {:dev\n> ;; so gen-class stuff works in the repl\n> {:aot [flambo.function ict.core]}\n> :provided\n> {:dependencies\n> [[org.apache.spark/spark-core_2.10 \"1.3.1\"]\n> [org.apache.spark/spark-streaming_2.10 \"1.3.1\"]\n> [org.apache.spark/spark-streaming-kafka_2.10 \"1.3.1\"]\n> [org.apache.spark/spark-streaming-flume_2.10 \"1.3.1\"]\n> [org.apache.spark/spark-sql_2.10 \"1.3.1\"]\n> [org.clojure/tools.nrepl \"0.2.7\"]]}\n> :uberjar\n> {:aot :all}})\n>\n> why are you putting flambo in :resource-paths? I don't believe it will be\n> included in the uberjar this way. You need to declare it under\n> :dependencies. Specifying your :target-path is not usually necessary. I\n> don't think it would cause a problem, but I'd just remove it unless you\n> changing it from the default for some special reason.\n>\n> -\n> Reply to this email directly or view it on GitHub<\n> https://github.com/yieldbot/flambo/issues/78#issuecomment-124873201>.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/yieldbot/flambo/issues/78#issuecomment-124876577>.\n>\n\n—\nReply to this email directly or view it on GitHub<https://github.com/yieldbot/flambo/issues/78#issuecomment-124878179>.\n"}}