{"action":"created","issue":{"url":"https://api.github.com/repos/tresata/spark-scalding/issues/1","labels_url":"https://api.github.com/repos/tresata/spark-scalding/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/tresata/spark-scalding/issues/1/comments","events_url":"https://api.github.com/repos/tresata/spark-scalding/issues/1/events","html_url":"https://github.com/tresata/spark-scalding/issues/1","id":55502062,"number":1,"title":"cascading.tuple.TupleException: unable to read from input identifier: 'unknown'","user":{"login":"nikolovivan","id":3465575,"avatar_url":"https://avatars.githubusercontent.com/u/3465575?v=3","gravatar_id":"","url":"https://api.github.com/users/nikolovivan","html_url":"https://github.com/nikolovivan","followers_url":"https://api.github.com/users/nikolovivan/followers","following_url":"https://api.github.com/users/nikolovivan/following{/other_user}","gists_url":"https://api.github.com/users/nikolovivan/gists{/gist_id}","starred_url":"https://api.github.com/users/nikolovivan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikolovivan/subscriptions","organizations_url":"https://api.github.com/users/nikolovivan/orgs","repos_url":"https://api.github.com/users/nikolovivan/repos","events_url":"https://api.github.com/users/nikolovivan/events{/privacy}","received_events_url":"https://api.github.com/users/nikolovivan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-26T16:20:00Z","updated_at":"2015-01-26T16:47:30Z","closed_at":null,"body":"I've been trying to find my way around this issue, but I have no luck. I cannot even run the demo on my cluster. I follow the steps exactly as explained and I run my command as follows:\r\n\r\n    spark-submit --class com.twitter.scalding.Tool --master yarn-client /home/inikolov/demo.jar com.tresata.spark.scalding.demo.DemoJob --hdfs --input /user/inikolov/fruits.bsv --output /user/inikolov/test\r\n\r\nI've tried prepending `hdfs://` to the paths as well, but the outcome is still the same.\r\n\r\nAmong the things I've tried was to run my custom code that reads SequenceFiles rather than just a text file, I've played with excluding dependencies, forcing dependency versions, etc. to what my cluster is running. Running in local mode also doesn't help.\r\n\r\nPlease find below some more info about my current set up:\r\n1. Spark version: spark-1.1.1-bin-hadoop2.3.\r\n2. Hadoop version: 2.3.0-cdh5.0.3.\r\n3. In my sequence file example the files have been generated with **0.11.1-cdh5** version of scalding, so I've tried forcing this one for my job.\r\n\r\nHere is a bit more of the stack trace I see:\r\n\r\n    15/01/26 16:03:04 ERROR TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job\r\n    15/01/26 16:03:04 INFO YarnClientClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool\r\n    15/01/26 16:03:04 INFO YarnClientClusterScheduler: Cancelling stage 1\r\n    15/01/26 16:03:04 INFO DAGScheduler: Failed to run runJob at CascadingRDD.scala:161\r\n    Exception in thread \"main\" java.lang.Throwable: If you know what exactly caused this error, please consider contributing to GitHub via following link.\r\n    https://github.com/twitter/scalding/wiki/Common-Exceptions-and-possible-reasons#orgapachesparksparkexception\r\n\tat com.twitter.scalding.Tool$.main(Tool.scala:132)\r\n\tat com.twitter.scalding.Tool.main(Tool.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)\r\n\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)\r\n\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n    Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 3, bl-hadoop-p19.prodsec.bl.visualdna.com): cascading.tuple.TupleException: unable to read from input identifier: 'unknown'\r\n        cascading.tuple.TupleEntrySchemeIterator.hasNext(TupleEntrySchemeIterator.java:127)\r\n        scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)\r\n        scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n        scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n        scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n        scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n        org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:144)\r\n        org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:58)\r\n        org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:55)\r\n        org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)\r\n        org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\r\n        org.apache.spark.scheduler.Task.run(Task.scala:54)\r\n        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)\r\n        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        java.lang.Thread.run(Thread.java:745)\r\n    Driver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)\r\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)\r\n\tat scala.Option.foreach(Option.scala:236)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:456)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:219)\r\n\tat akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\n\r\n\r\nI've looked around the source code, I've tried changing things, but nothing has changed for me. I would be really thankful if you could give me any pointers at what else I could look into."},"comment":{"url":"https://api.github.com/repos/tresata/spark-scalding/issues/comments/71492523","html_url":"https://github.com/tresata/spark-scalding/issues/1#issuecomment-71492523","issue_url":"https://api.github.com/repos/tresata/spark-scalding/issues/1","id":71492523,"user":{"login":"koertkuipers","id":821284,"avatar_url":"https://avatars.githubusercontent.com/u/821284?v=3","gravatar_id":"","url":"https://api.github.com/users/koertkuipers","html_url":"https://github.com/koertkuipers","followers_url":"https://api.github.com/users/koertkuipers/followers","following_url":"https://api.github.com/users/koertkuipers/following{/other_user}","gists_url":"https://api.github.com/users/koertkuipers/gists{/gist_id}","starred_url":"https://api.github.com/users/koertkuipers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koertkuipers/subscriptions","organizations_url":"https://api.github.com/users/koertkuipers/orgs","repos_url":"https://api.github.com/users/koertkuipers/repos","events_url":"https://api.github.com/users/koertkuipers/events{/privacy}","received_events_url":"https://api.github.com/users/koertkuipers/received_events","type":"User","site_admin":false},"created_at":"2015-01-26T16:47:30Z","updated_at":"2015-01-26T16:47:30Z","body":"hey,\r\nthanks for reporting this.\r\n\r\nthe exception you are seeing in the task (cascading.tuple.TupleException: unable to read from input identifier: 'unknown') is thrown by TupleEntrySchemeIterator if another error occurred while getting the next tuple.\r\n\r\ncan you provide me with the full stacktrace of the exception in the task? or better, the full logs of the failed task?\r\nyou can email them to koert at tresata.com\r\n\r\nthanks"}}