{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/schemas/issues/390","labels_url":"https://api.github.com/repos/ga4gh/schemas/issues/390/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/schemas/issues/390/comments","events_url":"https://api.github.com/repos/ga4gh/schemas/issues/390/events","html_url":"https://github.com/ga4gh/schemas/issues/390","id":102588885,"number":390,"title":"thoughts on data model for tracking object provenance","user":{"login":"dglazer","id":5824733,"avatar_url":"https://avatars.githubusercontent.com/u/5824733?v=3","gravatar_id":"","url":"https://api.github.com/users/dglazer","html_url":"https://github.com/dglazer","followers_url":"https://api.github.com/users/dglazer/followers","following_url":"https://api.github.com/users/dglazer/following{/other_user}","gists_url":"https://api.github.com/users/dglazer/gists{/gist_id}","starred_url":"https://api.github.com/users/dglazer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dglazer/subscriptions","organizations_url":"https://api.github.com/users/dglazer/orgs","repos_url":"https://api.github.com/users/dglazer/repos","events_url":"https://api.github.com/users/dglazer/events{/privacy}","received_events_url":"https://api.github.com/users/dglazer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-23T05:13:55Z","updated_at":"2015-08-23T06:36:06Z","closed_at":null,"body":"*WARNING: long post below. I'm hoping to help us build a shared conceptual framework to guide ongoing API design, and given the underlying complexity and the number of different priors, don't know how to do it justice in fewer words.*\r\n\r\nMy mental model of the genomics space is that there’s a “provenance chain” connecting objects that are derived from each other. Our API manages both data and metadata for the ‘dry’ objects (e.g. readgroupsets), and could be extended to manage metadata for the ‘wet’ objects (e.g. tissue samples). We currently do very little to represent the relationship between objects -- this issue explores what it would take to do more.\r\n\r\nThese thoughts are inspired by recent ongoing discussion about data models, including:\r\n * @jeromekelleher, in #380, pointing out the value of a well-defined data model\r\n * @diekhans, commenting on that issue (https://github.com/ga4gh/schemas/issues/380#issuecomment-132248587) to point out the importance of provenance tracking\r\n * @lh3, in #383, starting a good conversation about how to improve part of our provenance tracking\r\n\r\n#### The Provenance Chain in the Real World\r\nThe overall chain starts with an organism, and for our purposes ends with variants. There are several links along the way, in each of which an input object is processed to generate an output object.\r\n\r\nHere’s a diagram representing a typical chain. (I’m far from an expert on the upstream ‘wet’ steps; the downstream ‘dry’ steps show the objects currently managed by our API.)\r\n```\r\ntissue source ⇒ tissue sample ⇒ prepped sample ⇒ rgset ⇒ aligned rgset ⇒ call-column\r\n\\-------------- wet stuff (atoms) ------------/    \\-------- dry stuff (bits) --------/\r\n```\r\n\r\nHere are some examples of the processing that happens at each step, which our API would need to manage to capture full provenance:\r\n* tissue source ==> tissue sample:\r\n  * how was this sample gathered? (date, tissue type, collection method, ...)\r\n* tissue sample ==> prepped sample\r\n  * how was this sample processed? (date, library prep, amplification, …)\r\n* prepped sample ==> unaligned readgroupset\r\n  * how was this readgroupset generated? (date, sequencer, settings, …)\r\n* unaligned readgroupset ==> aligned readgroupset\r\n  * how was this readgroupset aligned? (date, tools, versions, cmdline flags, …)\r\n* aligned readgroupset ==> call-column\r\n  * how were these variants called? (date, tools, versions, cmdline flags, …)\r\n\r\nNote that it’s possible to process each input object in different ways at different times, which means there can be many outputs for each input. That leads to one-to-many relationships, where there can (e.g.) be multiple unaligned readgroupsets for a single prepped sample, or multiple aligned readgroupsets for a single unaligned one.\r\n\r\n#### API Requirements to Represent the Provenance Chain\r\n**Object Definitions**\r\nTo capture the full provenance of each object, we need to represent:\r\n\r\n1. who’s my parent? (i.e. the object from which I was derived)\r\n  * typically represented with some sort of id scheme, such as:\r\n    * **ad hoc user-populated** text sitting in names or metadata (ugly, but often state of the art)\r\n    * **user-assigned ids** that must be unique within some scope, such as a variantset\r\n    * **server-generated ids** that are unique within a server instance\r\n    * **globally unique registry ids** that refer to external registries of some sort\r\n    * **globally unique content-based ids**, computed by content checksumming of some sort\r\n2. how was I generated? (i.e. the processing done on my parent to generate me)\r\n  * typically represented with metadata fields\r\n  * we can use pre-defined first-class records and/or free-form key-value pairs\r\n\r\n**Method Definitions**\r\nApplications that want to work with provenance might need to know:\r\n\r\n1. what’s the full provenance of this object?\r\n  * one natural way to support this request is to include the provenance in the object definition, so it’s returned by the object Get methods\r\n  * another possible way is to define new `GetProvenance(<object id>)` methods for some object types\r\n2. what are all the objects (of a given type) that came from this object?\r\n  * one way to support this request is to include optional “filter by parent” fields in the Search methods that return lists of objects\r\n  * another possible way is to define new `GetChildObjects(<object id>)` methods for some object types\r\n\r\n#### Drilling into API Design: A Sketch\r\n*Notes:*\r\n* *This section contains early, only partially fleshed out, personal opinions, included mostly to give a concrete example of how we could shape our API to reflect the data model discussed in this issue. We need to first reach agreement on the data model; then we can discuss the tradeoffs of different specific representations.*\r\n* *Many of the details here have been discussed at length by the Metadata TT. I’ve included a few concepts I first saw there, but have almost certainly left out many good ideas in this sketch.*\r\n\r\n**Object Definitions**\r\n\r\n1. Every object needs an id, so its children can unambiguously refer to it.\r\n  * every object should have a server-generated id, that’s unique within a server instance. We should use these ids as the primary means of joining children to parents. (We can also have query methods that optionally join by other attributes for convenience.)\r\n  * we should use external registry ids whenever well-supported registries exist, but we can’t count on them existing for all object types, and we don’t want to force users to register every object. Therefore many object types should have an optional ‘external id’ field, but it should never be required.\r\n  * content-based ids are awesome, but they’re tricky to get right, and may take some time to nail down for all objects. Therefore we should adopt them wherever we’re comfortable there’s a good answer (as there is for references), but not force them into use elsewhere.\r\n\r\n2. We should continue to use a mix of first-class fields and key-value arrays to describe how each object was derived from its parent.\r\n  * the details of which fields need to be first-class will be different for each object type\r\n  * in an ideal world the first-class fields would always be sufficient, but I don’t think we’ll get to that world quickly enough, and therefore should continue to support key-value arrays as an escape valve\r\n\r\n**Method Definitions**\r\n\r\n1. We should include full “how was I derived” info in the object, so it’s returned by every Get method.\r\n2. We should add parent-id fields to most Search methods, so you can limit the results to only the children of a specific parent.\r\n  * for example, `SearchReadGroupSetsRequest` should optionally include the id of the prepped sample from which the reads were sequenced\r\n  * in some cases we’ll want to include grandparents and great-grandparents, so for example we might want to get a list of all the calls in a particular `VariantSet` that, via several intermediate steps, came from the same living organism\r\n\r\n\r\n#### Note on *derived from* vs. *contains*\r\nIn addition to the *derived from* relationships discussed above, there are also *contains* relationships where some objects logically include sets of other objects. For example:\r\n* a dataset logically contains one or more readgroupsets and variantsets\r\n* a readgroupset logically contains one or more readgroups, each of which logically contains many reads -- see the header comment in reads.avdl for details\r\n* a variantset logically contains one or more columns of calls\r\n\r\nThose relationships are also important, but they’re distinct from the provenance chain. If we do want to discuss them, I suggest doing so in a separate thread. (Personally, I don’t think they need revisiting, at least for now -- they seem to be doing their job well.)"},"comment":{"url":"https://api.github.com/repos/ga4gh/schemas/issues/comments/133793750","html_url":"https://github.com/ga4gh/schemas/issues/390#issuecomment-133793750","issue_url":"https://api.github.com/repos/ga4gh/schemas/issues/390","id":133793750,"user":{"login":"lh3","id":480346,"avatar_url":"https://avatars.githubusercontent.com/u/480346?v=3","gravatar_id":"","url":"https://api.github.com/users/lh3","html_url":"https://github.com/lh3","followers_url":"https://api.github.com/users/lh3/followers","following_url":"https://api.github.com/users/lh3/following{/other_user}","gists_url":"https://api.github.com/users/lh3/gists{/gist_id}","starred_url":"https://api.github.com/users/lh3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lh3/subscriptions","organizations_url":"https://api.github.com/users/lh3/orgs","repos_url":"https://api.github.com/users/lh3/repos","events_url":"https://api.github.com/users/lh3/events{/privacy}","received_events_url":"https://api.github.com/users/lh3/received_events","type":"User","site_admin":false},"created_at":"2015-08-23T06:36:05Z","updated_at":"2015-08-23T06:36:05Z","body":"One quick comment. What is missing here is the \"sample\" computational people talk about daily. It at times includes multiple prep samples/libraries, so is different from prepped sample/library. The name of this \"sample\" is the one we see in BAM and VCF header. It is unique in a project and links all types of data in the project."}}