{"action":"opened","issue":{"url":"https://api.github.com/repos/BD2KGenomics/cgcloud/issues/32","labels_url":"https://api.github.com/repos/BD2KGenomics/cgcloud/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/BD2KGenomics/cgcloud/issues/32/comments","events_url":"https://api.github.com/repos/BD2KGenomics/cgcloud/issues/32/events","html_url":"https://github.com/BD2KGenomics/cgcloud/issues/32","id":85598323,"number":32,"title":"Spark can't find Hadoop","user":{"login":"beaunorgeot","id":9386672,"avatar_url":"https://avatars.githubusercontent.com/u/9386672?v=3","gravatar_id":"","url":"https://api.github.com/users/beaunorgeot","html_url":"https://github.com/beaunorgeot","followers_url":"https://api.github.com/users/beaunorgeot/followers","following_url":"https://api.github.com/users/beaunorgeot/following{/other_user}","gists_url":"https://api.github.com/users/beaunorgeot/gists{/gist_id}","starred_url":"https://api.github.com/users/beaunorgeot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beaunorgeot/subscriptions","organizations_url":"https://api.github.com/users/beaunorgeot/orgs","repos_url":"https://api.github.com/users/beaunorgeot/repos","events_url":"https://api.github.com/users/beaunorgeot/events{/privacy}","received_events_url":"https://api.github.com/users/beaunorgeot/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-05T18:21:55Z","updated_at":"2015-06-05T18:21:55Z","closed_at":null,"body":"Spark loads the default Hadoop parameters instead of taking them from the Hadoop config files.\r\nThis results in failed spark/adam jobs\r\nSetting HADOOP_HOME doesn't solve this problem\r\nBut setting spark/ADAM_OPTS does\r\nADAM_OPTS=\"--driver-class-path /opt/sparkbox/hadoop/etc/hadoop\"\r\nBelow are traces for the ADAM call that pointed out the bug, our replication of the error using the spark shell, and some troubleshooting to rule out other potential causes \r\n\r\nSPARK can’t find Hadoop\r\n\r\n/usr/bin/time /home/sparkbox/adam-distribution_2.10-0.17.0/bin/adam-submit transform hdfs://spark-master:8020/C835.HCC1143_BL.*.bam hdfs://spark-master:8020/C835.HCC1143_BL.4.adam\r\nSpark assembly has been built with Hive, including Datanucleus jars on classpath\r\n2015-06-03 22:54:13 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nException in thread \"main\" java.lang.IllegalArgumentException: Wrong FS: hdfs://spark-master:8020/C835.HCC1143_BL.*.bam, expected: file:///\r\n\tat org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:79)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:506)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)\r\n\tat org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)\r\n\tat org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1411)\r\n\tat org.bdgenomics.adam.rdd.ADAMContext.loadBam(ADAMContext.scala:237)\r\n\tat org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadAlignments$1.apply(ADAMContext.scala:516)\r\n\tat org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadAlignments$1.apply(ADAMContext.scala:513)\r\n\tat org.apache.spark.rdd.Timer.time(Timer.scala:57)\r\n\tat org.bdgenomics.adam.rdd.ADAMContext.loadAlignments(ADAMContext.scala:511)\r\n\tat org.bdgenomics.adam.cli.Transform.run(Transform.scala:148)\r\n\tat org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:53)\r\n\tat org.bdgenomics.adam.cli.Transform.run(Transform.scala:85)\r\n\tat org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:105)\r\n\tat org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n\r\n\r\n_____\r\nWe can replicate the problem in the spark shell\r\n\r\nscala> fs.isDirectory(new Path(new URI(\"hdfs://spark-master:8020/C835.HCC1143_BL.4.bam\")))\r\njava.lang.IllegalArgumentException: Wrong FS: hdfs://spark-master:8020/C835.HCC1143_BL.4.bam, expected: file:///\r\n\tat org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:79)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:506)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)\r\n\tat org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)\r\n\tat org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1411)\r\n\tat $iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\r\n\tat $iwC$$iwC$$iwC.<init>(<console>:22)\r\n\tat $iwC$$iwC.<init>(<console>:24)\r\n\tat $iwC.<init>(<console>:26)\r\n\tat <init>(<console>:28)\r\n\tat .<init>(<console>:32)\r\n\tat .<clinit>(<console>)\r\n\tat .<init>(<console>:7)\r\n\tat .<clinit>(<console>)\r\n\tat $print(<console>)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)\r\n\tat org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)\r\n\tat org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)\r\n\tat org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)\r\n\tat org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)\r\n\tat org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828)\r\n\tat org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873)\r\n\tat org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785)\r\n\tat org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628)\r\n\tat org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636)\r\n\tat org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)\r\n\tat org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)\r\n\tat scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)\r\n\tat org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916)\r\n\tat org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011)\r\n\tat org.apache.spark.repl.Main$.main(Main.scala:31)\r\n\tat org.apache.spark.repl.Main.main(Main.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n\r\n\r\nscala> FileSystem.getDefaultUri(sc.hadoopConfiguration)\r\nres4: java.net.URI = file:///\r\n\r\n__\r\nSetting HADOOP_HOME doesn’t fix\r\n\r\nHADOOP_HOME=/opt/sparkbox/hadoop /usr/bin/time /home/sparkbox/adam-distribution_2.10-0.17.0/bin/adam-submit transform /C835.HCC1143_BL.4.bam hdfs://spark-master:8020/C835.HCC1143_BL.4.adam\r\nSpark assembly has been built with Hive, including Datanucleus jars on classpath\r\n2015-06-03 22:58:08 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nException in thread \"main\" java.lang.NullPointerException\r\n\r\nThe problem wasn't just how the hdfs path was being interpreted\r\n\r\nADAM_OPTS=/usr/bin/time /home/sparkbox/adam-distribution_2.10-0.17.0/bin/adam-submit transform /C835.HCC1143_BL.4.bam hdfs://spark-master:8020/C835.HCC1143_BL.4.adam\r\n\r\nSpark assembly has been built with Hive, including Datanucleus jars on classpath\r\njava.lang.ClassNotFoundException: org.bdgenomics.adam.cli.ADAMMain\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:366)\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:355)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:354)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:425)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:358)\r\n\tat java.lang.Class.forName0(Native Method)\r\n\tat java.lang.Class.forName(Class.java:278)\r\n\tat org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n_________________\r\n\r\n\r\n\r\n"}}