{"action":"opened","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/14","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/14/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/14/events","html_url":"https://github.com/GoogleCloudPlatform/bigdata-interop/issues/14","id":112389211,"number":14,"title":"\"java.io.IOException: Not an Avro data file\" when using AvroBigQueryInputFormat","user":{"login":"nevillelyh","id":468514,"avatar_url":"https://avatars.githubusercontent.com/u/468514?v=3","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-20T15:05:16Z","updated_at":"2015-10-20T15:05:16Z","closed_at":null,"body":"Code looks like this\r\n```scala\r\nval conf = self.hadoopConfiguration\r\n\r\nconf.setClass(\r\n  AbstractBigQueryInputFormat.INPUT_FORMAT_CLASS_KEY,\r\n  classOf[AvroBigQueryInputFormat], classOf[InputFormat[_, _]])\r\nBigQueryConfiguration.configureBigQueryInput(conf, table.getProjectId, table.getDatasetId, table.getTableId)\r\n\r\nsc.newAPIHadoopFile(\r\n  BigQueryStrings.toString(table).replace(':', '.'),\r\n  classOf[AvroBigQueryInputFormat],\r\n  classOf[LongWritable], classOf[GenericData.Record]).map(_._2)\r\n```\r\n\r\nStacktrace:\r\n```\r\nCaused by: java.io.IOException: Not an Avro data file\r\n\tat org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50)\r\n\tat com.google.cloud.hadoop.io.bigquery.AvroRecordReader.initialize(AvroRecordReader.java:49)\r\n\tat com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader.nextKeyValue(DynamicFileListRecordReader.java:176)\r\n\tat org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:143)\r\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:277)\r\n\tat org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)\r\n\tat org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:262)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n```\r\n\r\nI saw these in the log:\r\n`15/10/20 14:51:54 INFO bigquery.AbstractBigQueryInputFormat: Resolved GCS export path: 'gs://starship/hadoop/tmp/bigquery/job_201510201451_0002'\r\n15/10/20 14:51:55 INFO bigquery.ShardedExportToCloudStorage: Computed '2' shards for sharded BigQuery export.\r\n15/10/20 14:51:55 INFO bigquery.ShardedExportToCloudStorage: Table 'prefab-wave-844:bigquery_staging.spark_query_20151020144723_1201584542' to be exported has 4251779 rows and 1641186694 bytes\r\n15/10/20 14:51:55 INFO bigquery.ShardedExportToCloudStorage: Computed '2' shards for sharded BigQuery export.\r\n15/10/20 14:51:55 INFO bigquery.ShardedExportToCloudStorage: Table 'prefab-wave-844:bigquery_staging.spark_query_20151020144723_1201584542' to be exported has 4251779 rows and 1641186694 bytes\r\n`\r\nAnd verified that the export path indeed contains Avro files only and nothing else."}}