{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/server/issues/134","labels_url":"https://api.github.com/repos/ga4gh/server/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/134/comments","events_url":"https://api.github.com/repos/ga4gh/server/issues/134/events","html_url":"https://github.com/ga4gh/server/pull/134","id":56965889,"number":134,"title":"Datasets","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-02-08T20:19:58Z","updated_at":"2015-02-09T16:18:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ga4gh/server/pulls/134","html_url":"https://github.com/ga4gh/server/pull/134","diff_url":"https://github.com/ga4gh/server/pull/134.diff","patch_url":"https://github.com/ga4gh/server/pull/134.patch"},"body":"This is a very rough work-in-progress for adding Dataset handling. This should probably go before #121, as there are some fairly fundamental and disruptive changes proposed here. This is just a sketch so we can discuss the ideas more concretely, so I'd really appreciate any feedback people might have here. Here's major changes:\r\n\r\n1. We change the server CLI to take a single positional argument which is the config file. Within the config file we have a list DATASETS, which contains a list of directories. Each directory is assumed to correspond to exactly one dataset, and has a prescribed format. For now, we just have a 'variants' sub directory, but we will also have a 'reads' directory too.\r\n2. Within the variants sub-directory, each directory corresponds to a VariantSet. For now, we inspect the directory to decide whether we should allocate a WormtableVariantSet or a TabixVariantSet. I've outlined what I think is a better idea in the comments based on 'Shards' of the variant set.\r\n3. I've introduced the concept of a `DataObject` which should be the superclass of all objects that represent an aggregation of data in some way. So, Dataset, VariantSet, ReadSet, ReferenceSet, all of these should be DataObjects. Hopefully we can push a lot of the tedious housekeeping and file system walking code up here. Hopefully this will help make testing a bit easier also.  I think we should explicitly model this as a hierarchy, as this might make the tricky iteration/page token problems a bit simpler, as we can then view it as a tree-traversal problem and maybe generalise the code.\r\n\r\nFeedback appreciated!"},"comment":{"url":"https://api.github.com/repos/ga4gh/server/issues/comments/73537254","html_url":"https://github.com/ga4gh/server/pull/134#issuecomment-73537254","issue_url":"https://api.github.com/repos/ga4gh/server/issues/134","id":73537254,"user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"created_at":"2015-02-09T16:18:46Z","updated_at":"2015-02-09T16:18:46Z","body":"My understanding (and I could be completely wrong) is that the dataset is a top level container for read and variant data. So we'd have\r\n```\r\ndataset1/\r\n       + readGroups\r\n             + readGroupSet1\r\n             + readGroupSet2\r\n             + ....\r\n       + variantSets\r\n             + variantSet1\r\n             + ....\r\ndataset2/\r\n       + ....\r\n```\r\nThe dataset container is then used to determine access privileges --- is user X entitled to read from dataset Y?"}}