{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-avro/issues/91","labels_url":"https://api.github.com/repos/databricks/spark-avro/issues/91/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-avro/issues/91/comments","events_url":"https://api.github.com/repos/databricks/spark-avro/issues/91/events","html_url":"https://github.com/databricks/spark-avro/issues/91","id":107837034,"number":91,"title":"Spark-avro Fails To Write DF in EMR 4.0","user":{"login":"nadersalehi","id":11711964,"avatar_url":"https://avatars.githubusercontent.com/u/11711964?v=3","gravatar_id":"","url":"https://api.github.com/users/nadersalehi","html_url":"https://github.com/nadersalehi","followers_url":"https://api.github.com/users/nadersalehi/followers","following_url":"https://api.github.com/users/nadersalehi/following{/other_user}","gists_url":"https://api.github.com/users/nadersalehi/gists{/gist_id}","starred_url":"https://api.github.com/users/nadersalehi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nadersalehi/subscriptions","organizations_url":"https://api.github.com/users/nadersalehi/orgs","repos_url":"https://api.github.com/users/nadersalehi/repos","events_url":"https://api.github.com/users/nadersalehi/events{/privacy}","received_events_url":"https://api.github.com/users/nadersalehi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-23T02:04:12Z","updated_at":"2015-09-23T03:38:12Z","closed_at":null,"body":"I have a simple code which works without a problem when running on my laptop (which uses pre-built spark-1.4.1 hadoop 2.4).  When running the same code in EMR 4.0, the code crashes when it tries to execute the following line\r\n\r\n```\r\ntime_series_df.write.format('com.databricks.spark.avro').save(args.output_dir)\r\n```\r\nStack trace is provided below:\r\n\r\n> 2015-09-23 01:59:42,559 ERROR [Thread-3] sources.DefaultWriterContainer (Logging.scala:logError(75)) - Job job_201509230159_0000 aborted.\r\n> Traceback (most recent call last):\r\n>   File \"/home/hadoop/./time_series_producer.py\", line 336, in <module>\r\n>     main()\r\n>   File \"/home/hadoop/./time_series_producer.py\", line 333, in main\r\n>     .save(args.output_dir)\r\n>   File \"/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py\", line 304, in save\r\n>   File \"/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py\", line 538, in __call__\r\n>   File \"/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py\", line 300, in get_return_value\r\n> py4j.protocol.Py4JJavaError: An error occurred while calling o171.save.\r\n> : org.apache.spark.SparkException: Job aborted.\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.insert(commands.scala:166)\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.run(commands.scala:139)\r\n> \tat org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)\r\n> \tat org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)\r\n> \tat org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)\r\n> \tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)\r\n> \tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)\r\n> \tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)\r\n> \tat org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)\r\n> \tat org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950)\r\n> \tat org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950)\r\n> \tat org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:336)\r\n> \tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144)\r\n> \tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:135)\r\n> \tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n> \tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n> \tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n> \tat java.lang.reflect.Method.invoke(Method.java:606)\r\n> \tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)\r\n> \tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)\r\n> \tat py4j.Gateway.invoke(Gateway.java:259)\r\n> \tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)\r\n> \tat py4j.commands.CallCommand.execute(CallCommand.java:79)\r\n> \tat py4j.GatewayConnection.run(GatewayConnection.java:207)\r\n> \tat java.lang.Thread.run(Thread.java:745)\r\n> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 11.0 failed 4 times, most recent failure: Lost task 1.3 in stage 11.0 (TID 71, ip-172-31-36-39.ec2.internal): org.apache.spark.SparkException: Task failed while writing rows.\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows$1(commands.scala:191)\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:160)\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:160)\r\n> \tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)\r\n> \tat org.apache.spark.scheduler.Task.run(Task.scala:70)\r\n> \tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)\r\n> \tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n> \tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n> \tat java.lang.Thread.run(Thread.java:745)\r\n> Caused by: java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;\r\n> \tat org.apache.avro.mapreduce.AvroKeyRecordWriter.<init>(AvroKeyRecordWriter.java:55)\r\n> \tat org.apache.avro.mapreduce.AvroKeyOutputFormat$RecordWriterFactory.create(AvroKeyOutputFormat.java:79)\r\n> \tat org.apache.avro.mapreduce.AvroKeyOutputFormat.getRecordWriter(AvroKeyOutputFormat.java:105)\r\n> \tat com.databricks.spark.avro.AvroOutputWriter.<init>(AvroOutputWriter.scala:82)\r\n> \tat com.databricks.spark.avro.AvroOutputWriterFactory.newInstance(AvroOutputWriterFactory.scala:31)\r\n> \tat org.apache.spark.sql.sources.DefaultWriterContainer.initWriters(commands.scala:470)\r\n> \tat org.apache.spark.sql.sources.BaseWriterContainer.executorSideSetup(commands.scala:360)\r\n> \tat org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows$1(commands.scala:172)\r\n> \t... 8 more\r\n> \r\n> Driver stacktrace:\r\n> \tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273)\r\n> \tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264)\r\n> \tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263)\r\n> \tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n> \tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\r\n> \tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263)\r\n> \tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)\r\n> \tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)\r\n> \tat scala.Option.foreach(Option.scala:236)\r\n> \tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)\r\n> \tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457)\r\n> \tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418)\r\n> \tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\r\n> \r\n> 2015-09-23 01:59:42,591 INFO  [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logInfo(59)) - Lost task 5.3 in stage 11.0 (TID 76) on executor ip-172-31-36-38.ec2.internal: org.apache.spark.SparkException (Task failed while writing rows.) [duplicate 28]\r\n"},"comment":{"url":"https://api.github.com/repos/databricks/spark-avro/issues/comments/142481503","html_url":"https://github.com/databricks/spark-avro/issues/91#issuecomment-142481503","issue_url":"https://api.github.com/repos/databricks/spark-avro/issues/91","id":142481503,"user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T03:38:12Z","updated_at":"2015-09-23T03:38:12Z","body":"Also, just to clarify: which version of `spark-avro` are you using?"}}