{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/15","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/15/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/15/events","html_url":"https://github.com/GoogleCloudPlatform/bigdata-interop/issues/15","id":112461871,"number":15,"title":"ShardedExportToCloudStorage should allow custom splits","user":{"login":"nevillelyh","id":468514,"avatar_url":"https://avatars.githubusercontent.com/u/468514?v=3","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-20T21:12:03Z","updated_at":"2015-10-23T16:36:47Z","closed_at":null,"body":"Looks like the # of splits is determined by # of shards. This should be fine for traditional M/R jobs but in Spark where there are a fixed number of executors, it might be better to use all of them even if # of executors > # shards.\r\n\r\nhttps://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/bigquery/src/main/java/com/google/cloud/hadoop/io/bigquery/ShardedExportToCloudStorage.java#L73"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/comments/150627705","html_url":"https://github.com/GoogleCloudPlatform/bigdata-interop/issues/15#issuecomment-150627705","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/bigdata-interop/issues/15","id":150627705,"user":{"login":"nevillelyh","id":468514,"avatar_url":"https://avatars.githubusercontent.com/u/468514?v=3","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T16:36:47Z","updated_at":"2015-10-23T16:36:47Z","body":"Here you go. Thanks for looking!\r\n```\r\nCaused by: java.lang.IllegalArgumentException: Split should be instance of UnshardedInputSplit.\r\n\tat com.google.cloud.hadoop.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)\r\n\tat com.google.cloud.hadoop.io.bigquery.AbstractBigQueryInputFormat.createRecordReader(AbstractBigQueryInputFormat.java:158)\r\n\tat com.google.cloud.hadoop.io.bigquery.AbstractBigQueryInputFormat.createRecordReader(AbstractBigQueryInputFormat.java:145)\r\n\tat org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:131)\r\n\tat org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)\r\n\tat org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\r\n\tat org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:262)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n\t... 3 more\r\n```"}}