{"action":"created","comment":{"url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/comments/37790904","id":37790904,"diff_hunk":"@@ -0,0 +1,74 @@\n+package org.broadinstitute.hellbender.engine.spark;\n+\n+import com.google.common.collect.Lists;\n+import org.apache.spark.api.java.JavaPairRDD;\n+import org.apache.spark.api.java.JavaRDD;\n+import org.broadinstitute.hellbender.engine.dataflow.datasources.ReferenceDataflowSource;\n+import org.broadinstitute.hellbender.engine.dataflow.datasources.ReferenceShard;\n+import org.broadinstitute.hellbender.utils.SimpleInterval;\n+import org.broadinstitute.hellbender.utils.read.GATKRead;\n+import org.broadinstitute.hellbender.utils.reference.ReferenceBases;\n+import scala.Tuple2;\n+\n+import java.util.List;\n+\n+/**\n+ * RefBasesForReads queries the Google Genomics API for reference bases overlapping all of the reads.\n+ *\n+ * step 1: key reads by reference shards\n+ *\n+ * |--------- shard 0 ----------|---------- shard 1 ----------|--------- shard 2 ----------|--------- shard 3 ----------|\n+ *           |------ read a -----|                             |-- read b --|   |---- read c ----|\n+ *\n+ * step 2: group reads by the shard they start in\n+ *\n+ *  |--------- shard 0 ----------|\n+ *            |------ read a -----|\n+ *\n+ *\n+ *  |--------- shard 2 ----------|\n+ *   |-- read b --|   |---- read c ----|\n+ *\n+ *  step 3: query the Google Genomics API for all bases needed for each shard\n+ *\n+\n+ * |--- ref bases 1 ---|        |--------- ref bases 2 -----------|\n+ * |------ read a -----|        |-- read b --|   |---- read c ----|\n+ *\n+ *  step 4: pair the ref bases needed for each read with the read\n+ *\n+ * |------ read a -----|        |-- read b --|   |---- read c ----|\n+ * |-- ref bases 1a ---|        |ref bases 2b|   |- ref bases 2c -|\n+ *\n+ * or in code,\n+ *  KV<read a, ref bases 1a>\n+ *  KV<read b, ref bases 2b>\n+ *  KV<read c, ref bases 2c>\n+ *\n+ * The reference bases paired with each read can be customized by passing in a reference window function\n+ * inside the {@link ReferenceDataflowSource} argument to {@link #addBases}. See {@link org.broadinstitute.hellbender.engine.dataflow.datasources.RefWindowFunctions} for examples.","path":"src/main/java/org/broadinstitute/hellbender/engine/spark/JoinReadsWithRefBases.java","position":49,"original_position":49,"commit_id":"2310062d70e1531ee2dc79556797d39ea1be1807","original_commit_id":"2310062d70e1531ee2dc79556797d39ea1be1807","user":{"login":"droazen","id":798637,"avatar_url":"https://avatars.githubusercontent.com/u/798637?v=3","gravatar_id":"","url":"https://api.github.com/users/droazen","html_url":"https://github.com/droazen","followers_url":"https://api.github.com/users/droazen/followers","following_url":"https://api.github.com/users/droazen/following{/other_user}","gists_url":"https://api.github.com/users/droazen/gists{/gist_id}","starred_url":"https://api.github.com/users/droazen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/droazen/subscriptions","organizations_url":"https://api.github.com/users/droazen/orgs","repos_url":"https://api.github.com/users/droazen/repos","events_url":"https://api.github.com/users/droazen/events{/privacy}","received_events_url":"https://api.github.com/users/droazen/received_events","type":"User","site_admin":false},"body":"(If this feature is not supported yet, you should add a comment to that effect)","created_at":"2015-08-24T19:05:06Z","updated_at":"2015-08-24T19:05:06Z","html_url":"https://github.com/broadinstitute/hellbender/pull/850#discussion_r37790904","pull_request_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850","_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/comments/37790904"},"html":{"href":"https://github.com/broadinstitute/hellbender/pull/850#discussion_r37790904"},"pull_request":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850"}}},"pull_request":{"url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850","id":43057494,"html_url":"https://github.com/broadinstitute/hellbender/pull/850","diff_url":"https://github.com/broadinstitute/hellbender/pull/850.diff","patch_url":"https://github.com/broadinstitute/hellbender/pull/850.patch","issue_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/850","number":850,"state":"open","locked":false,"title":"Initial Spark dataflow commit, basic skeleton with some tests.","user":{"login":"davidadamsphd","id":12084283,"avatar_url":"https://avatars.githubusercontent.com/u/12084283?v=3","gravatar_id":"","url":"https://api.github.com/users/davidadamsphd","html_url":"https://github.com/davidadamsphd","followers_url":"https://api.github.com/users/davidadamsphd/followers","following_url":"https://api.github.com/users/davidadamsphd/following{/other_user}","gists_url":"https://api.github.com/users/davidadamsphd/gists{/gist_id}","starred_url":"https://api.github.com/users/davidadamsphd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidadamsphd/subscriptions","organizations_url":"https://api.github.com/users/davidadamsphd/orgs","repos_url":"https://api.github.com/users/davidadamsphd/repos","events_url":"https://api.github.com/users/davidadamsphd/events{/privacy}","received_events_url":"https://api.github.com/users/davidadamsphd/received_events","type":"User","site_admin":false},"body":"A few things worth mentioning.\r\n\r\n1) This skeleton is mostly a direct port of the existing Dataflow pipeline.\r\n2) I had to modify some test data because there is a (masked) bug in the Dataflow code, see #795.\r\n3) Serialization was a slight pain and I had to bump the kryo version to the latest 2.x, as well as add two custom Serializers. If there's a cleaner way to handle any of that I'm all ears.\r\n4) I'm using Hadoop-bam for reading and writing reads and variants.\r\n5) There are unit tests for all code except for the skeleton itself. I've run it on a cluster on my machine successfully, but haven't written the tests (which will take a little while to design and get right).\r\n","created_at":"2015-08-21T16:45:48Z","updated_at":"2015-08-24T19:05:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2c867f3918a4ad97581315ea7a00e58c221dc648","assignee":{"login":"tomwhite","id":85085,"avatar_url":"https://avatars.githubusercontent.com/u/85085?v=3","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850/commits","review_comments_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850/comments","review_comment_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls/comments{/number}","comments_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/850/comments","statuses_url":"https://api.github.com/repos/broadinstitute/hellbender/statuses/2310062d70e1531ee2dc79556797d39ea1be1807","head":{"label":"broadinstitute:da_spark_skeleton","ref":"da_spark_skeleton","sha":"2310062d70e1531ee2dc79556797d39ea1be1807","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":27452807,"name":"hellbender","full_name":"broadinstitute/hellbender","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/hellbender","description":"GATK4 development","fork":false,"url":"https://api.github.com/repos/broadinstitute/hellbender","forks_url":"https://api.github.com/repos/broadinstitute/hellbender/forks","keys_url":"https://api.github.com/repos/broadinstitute/hellbender/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/hellbender/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/hellbender/teams","hooks_url":"https://api.github.com/repos/broadinstitute/hellbender/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/hellbender/events","assignees_url":"https://api.github.com/repos/broadinstitute/hellbender/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/hellbender/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/hellbender/tags","blobs_url":"https://api.github.com/repos/broadinstitute/hellbender/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/hellbender/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/hellbender/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/hellbender/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/hellbender/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/hellbender/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/hellbender/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/hellbender/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/hellbender/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/hellbender/subscription","commits_url":"https://api.github.com/repos/broadinstitute/hellbender/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/hellbender/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/hellbender/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/hellbender/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/hellbender/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/hellbender/merges","archive_url":"https://api.github.com/repos/broadinstitute/hellbender/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/hellbender/downloads","issues_url":"https://api.github.com/repos/broadinstitute/hellbender/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/hellbender/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/hellbender/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/hellbender/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/hellbender/releases{/id}","created_at":"2014-12-02T20:50:36Z","updated_at":"2015-08-12T03:14:08Z","pushed_at":"2015-08-24T17:25:20Z","git_url":"git://github.com/broadinstitute/hellbender.git","ssh_url":"git@github.com:broadinstitute/hellbender.git","clone_url":"https://github.com/broadinstitute/hellbender.git","svn_url":"https://github.com/broadinstitute/hellbender","homepage":"","size":39106,"stargazers_count":29,"watchers_count":29,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":242,"forks":8,"open_issues":242,"watchers":29,"default_branch":"master"}},"base":{"label":"broadinstitute:master","ref":"master","sha":"921e730af17d0461ee7f04512f91906a9235165c","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":27452807,"name":"hellbender","full_name":"broadinstitute/hellbender","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/hellbender","description":"GATK4 development","fork":false,"url":"https://api.github.com/repos/broadinstitute/hellbender","forks_url":"https://api.github.com/repos/broadinstitute/hellbender/forks","keys_url":"https://api.github.com/repos/broadinstitute/hellbender/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/hellbender/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/hellbender/teams","hooks_url":"https://api.github.com/repos/broadinstitute/hellbender/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/hellbender/events","assignees_url":"https://api.github.com/repos/broadinstitute/hellbender/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/hellbender/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/hellbender/tags","blobs_url":"https://api.github.com/repos/broadinstitute/hellbender/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/hellbender/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/hellbender/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/hellbender/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/hellbender/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/hellbender/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/hellbender/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/hellbender/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/hellbender/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/hellbender/subscription","commits_url":"https://api.github.com/repos/broadinstitute/hellbender/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/hellbender/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/hellbender/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/hellbender/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/hellbender/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/hellbender/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/hellbender/merges","archive_url":"https://api.github.com/repos/broadinstitute/hellbender/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/hellbender/downloads","issues_url":"https://api.github.com/repos/broadinstitute/hellbender/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/hellbender/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/hellbender/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/hellbender/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/hellbender/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/hellbender/releases{/id}","created_at":"2014-12-02T20:50:36Z","updated_at":"2015-08-12T03:14:08Z","pushed_at":"2015-08-24T17:25:20Z","git_url":"git://github.com/broadinstitute/hellbender.git","ssh_url":"git@github.com:broadinstitute/hellbender.git","clone_url":"https://github.com/broadinstitute/hellbender.git","svn_url":"https://github.com/broadinstitute/hellbender","homepage":"","size":39106,"stargazers_count":29,"watchers_count":29,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":242,"forks":8,"open_issues":242,"watchers":29,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850"},"html":{"href":"https://github.com/broadinstitute/hellbender/pull/850"},"issue":{"href":"https://api.github.com/repos/broadinstitute/hellbender/issues/850"},"comments":{"href":"https://api.github.com/repos/broadinstitute/hellbender/issues/850/comments"},"review_comments":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850/comments"},"review_comment":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/broadinstitute/hellbender/pulls/850/commits"},"statuses":{"href":"https://api.github.com/repos/broadinstitute/hellbender/statuses/2310062d70e1531ee2dc79556797d39ea1be1807"}}}}