{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/server/issues/44","labels_url":"https://api.github.com/repos/ga4gh/server/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/44/comments","events_url":"https://api.github.com/repos/ga4gh/server/issues/44/events","html_url":"https://github.com/ga4gh/server/issues/44","id":51475762,"number":44,"title":"Proposed data directory layout","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ga4gh/server/labels/help+wanted","name":"help wanted","color":"159818"},{"url":"https://api.github.com/repos/ga4gh/server/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2014-12-09T20:15:07Z","updated_at":"2015-01-14T09:16:56Z","closed_at":null,"body":"Currently, we only support variant data derived from VCF, in either vcf.gz or wormtable format. We need to support other types of data as we move to support the full API. I suggest we use a designated file system layout for the data directory to discover the files and their relationships. An initial pass at this might be\r\n```\r\ndataDir/\r\n     datasetId1/\r\n           variants/\r\n                 variantSetId1/\r\n                       chrom1.vcf.gz\r\n                       chrom2.vcf.gz\r\n                 variantSetId2/\r\n                       variants.wt\r\n           reads/\r\n                  [Some directory structure laying out BAM/SAM files]\r\n           references/\r\n                  [Some directory structure laying out references]\r\n     dataSetId2/\r\n           variants/\r\n                  ....\r\n           reads/\r\n                  ....\r\n           references\r\n```  \r\nThat is, within the data directory, each directory `d` corresponds to a `Dataset` with ID `d`. Within a given dataset, we have three directories, called `variants`, `reads` and `references`. Within the `variants` directory, we then have more directories, one for each `VariantSet`. Within a `VariantSet` directory, we have a bunch of files that define these variants in what ever formats we support. \r\n\r\nFor data within a given variant set, we could follow the existing convention used for the TabixBackend: we can split the variant set over as many files as we like, so long as we don't have data for the same chromosome in two files. I've deliberately mixed up wormtable and VCF format files in here. The input should detect the file type and make an instance of the appropriate handler based on the extension. If we don't support a given file type then we complain and quit. This would get rid of the currently very clumsy model of specifying `wormtable` or `tabix` as command line arguments to determine the type of backend to allocate.\r\n\r\nWe'd then just call the server using ``ga4gh_server dataDir`` for development and testing purposes. For deployment an absolute path to the required data directory in a config file would be all that is needed.\r\n\r\nThe idea is that we use the file system hierarchy to explicitly set out the structure of the data to be served. This keeps the number of configuration variables we need as we deal with more complex datasets down. The downside is that we do require a pretty complex file system hierarchy to specify the data to be served. One way we could alleviate this problem is by providing a tool to check the layout of the hierarchy and to raise errors if it's not well formed. This could also print some summaries about the data that's in there too, so that people can get an overview of what they're about to serve.\r\n\r\nI've not filled out the reads and references section, as I don't understand this side of things all that well. Will this sort of decomposition make sense here too? Does this hierarchy capture the structure of the API, or is it too restrictive? Any feedback would be much appreciated!\r\n           \r\n"},"comment":{"url":"https://api.github.com/repos/ga4gh/server/issues/comments/69888653","html_url":"https://github.com/ga4gh/server/issues/44#issuecomment-69888653","issue_url":"https://api.github.com/repos/ga4gh/server/issues/44","id":69888653,"user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"created_at":"2015-01-14T09:16:56Z","updated_at":"2015-01-14T09:16:56Z","body":"Would you mind fleshing out your idea with an example of what the config file would look like @kerrydc? I'm struggling to imagine how this approach would scale to hundreds of files (which we definitely need to support to be useful), so it would be good to talk through a concrete example here."}}