{"action":"created","issue":{"url":"https://api.github.com/repos/bigdatagenomics/adam/issues/651","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/651/labels{/name}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/651/comments","events_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/651/events","html_url":"https://github.com/bigdatagenomics/adam/issues/651","id":68734399,"number":651,"title":"Support Hive-style partitioning","user":{"login":"tomwhite","id":85085,"avatar_url":"https://avatars.githubusercontent.com/u/85085?v=3","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-04-15T16:52:47Z","updated_at":"2015-04-16T17:16:05Z","closed_at":null,"body":"It's common to partition sequence data by locus. This change would make it possible to partition genotypes (alignments, etc) using a Hive-compatible directory structure like chr=M/pos=N (N is something like `floor(position/10^K)`).\r\n\r\nQuerying would then be more efficient since the SQL engine would only need to read files in the partitions of interest (typically one partition when doing a point query)."},"comment":{"url":"https://api.github.com/repos/bigdatagenomics/adam/issues/comments/93788235","html_url":"https://github.com/bigdatagenomics/adam/issues/651#issuecomment-93788235","issue_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/651","id":93788235,"user":{"login":"tomwhite","id":85085,"avatar_url":"https://avatars.githubusercontent.com/u/85085?v=3","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"created_at":"2015-04-16T17:16:05Z","updated_at":"2015-04-16T17:16:05Z","body":"Here's an initial go that uses Kite to do partitioning:\r\n\r\nhttps://github.com/bigdatagenomics/adam/compare/master...tomwhite:ADAM-651-hive-partitions\r\n\r\nThe idea is to add a partition command that takes a partition strategy file that defines which fields are used in constructing the partitions, and how the records map to partition values. For example, [this partition strategy](https://github.com/bigdatagenomics/adam/blob/df6355dfabed5c87f6076a06398deef30e11c129/adam-cli/src/test/resources/genotypes-partition-strategy.json) partitions by chromosome and bucketed position.\r\n\r\nThis is not quite ready yet, as it depends on a snapshot version of Kite for https://issues.cloudera.org/browse/CDK-986 and https://issues.cloudera.org/browse/CDK-988."}}