{"action":"created","issue":{"url":"https://api.github.com/repos/bigdatagenomics/adam/issues/890","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/890/labels{/name}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/890/comments","events_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/890/events","html_url":"https://github.com/bigdatagenomics/adam/issues/890","id":119371577,"number":890,"title":"How to filter genotype RDD with FeatureRDD","user":{"login":"NeillGibson","id":15608993,"avatar_url":"https://avatars.githubusercontent.com/u/15608993?v=3","gravatar_id":"","url":"https://api.github.com/users/NeillGibson","html_url":"https://github.com/NeillGibson","followers_url":"https://api.github.com/users/NeillGibson/followers","following_url":"https://api.github.com/users/NeillGibson/following{/other_user}","gists_url":"https://api.github.com/users/NeillGibson/gists{/gist_id}","starred_url":"https://api.github.com/users/NeillGibson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeillGibson/subscriptions","organizations_url":"https://api.github.com/users/NeillGibson/orgs","repos_url":"https://api.github.com/users/NeillGibson/repos","events_url":"https://api.github.com/users/NeillGibson/events{/privacy}","received_events_url":"https://api.github.com/users/NeillGibson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-11-29T17:26:57Z","updated_at":"2015-12-01T16:39:38Z","closed_at":null,"body":"Hi, \r\n\r\nHow can I filter a genotypeRDD with a FeatureRDD?  I get the following error:\r\n```\r\n(BroadcastRegionJoin.partitionAndJoin(genes, humanGenotypesRDD).map(_._2))\r\n<console>:39: error: type mismatch;\r\n found   : org.apache.spark.rdd.RDD[org.bdgenomics.formats.avro.Feature]\r\n required: org.apache.spark.rdd.RDD[(org.bdgenomics.adam.models.ReferenceRegion, ?)]\r\n              (BroadcastRegionJoin.partitionAndJoin(genesRDD, genotypesRDD).map(_._2))\r\n```\r\n\r\nwith this code:\r\n```\r\nimport org.bdgenomics.adam.rdd.ADAMContext\r\nimport org.bdgenomics.formats.avro._\r\nimport org.bdgenomics.adam.rdd.ADAMContext._\r\nimport org.bdgenomics.adam.rdd.BroadcastRegionJoin\r\nimport org.bdgenomics.adam.rdd.ADAMContext._\r\n\r\n # Load the genotype RDDs using the AdamContext\r\nval ac = new ADAMContext(sc)\r\nval genotypesRDD = ac.loadGenotypes(\"/user/ec2-user/1kg/chr22.adam\")\r\n\r\nval genesRDD = ac.loadFeatures(\"/user/ec2-user/1kg/gene_annotation.adam\")\r\n\r\n# based on example from Spark Analytics book (page 213 bottom)\r\ngenotypesInGenesRdd = (BroadcastRegionJoin.partitionAndJoin(genesRDD , genotypesRDD).map(_._2))\r\n\r\n```\r\n\r\nDo I need to convert the genotypeRDD and FeatureRDD to a ReferenceRegionRDD ? Is this an implicit conversion done automatically by importing a certain class?\r\n\r\nThank you, \r\n\r\nNeill"},"comment":{"url":"https://api.github.com/repos/bigdatagenomics/adam/issues/comments/161025464","html_url":"https://github.com/bigdatagenomics/adam/issues/890#issuecomment-161025464","issue_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/890","id":161025464,"user":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=3","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T16:39:38Z","updated_at":"2015-12-01T16:39:38Z","body":"It shouldn't be terribly expensive, however you may want to try running the ShuffleRegionJoin instead of the BroadcastRegionJoin. I'll need to look over the GTF loading code, but I'm not sure off of the top of my head whether that code maps each feature in a GTF to a `Feature` object, or just the Gene/Transcript. The reason I'm concerned about that is that if you have just genes as Features, you'll have an RDD of O(20k) `Feature`s, while if you have all GTF features (gene, transcript, exon, intron, cds, etc), it could be several orders of magnitude bigger and thus might not fit on the driver, which would cause the BroadcastRegionJoin to fail where the ShuffleRegionJoin would succeed. That being said, I'm a bit skeptical of that as the problem. Are you joining against the whole 1000Genomes variant calls? I can try to repro this on my side."}}