{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/schemas/issues/395","labels_url":"https://api.github.com/repos/ga4gh/schemas/issues/395/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/schemas/issues/395/comments","events_url":"https://api.github.com/repos/ga4gh/schemas/issues/395/events","html_url":"https://github.com/ga4gh/schemas/pull/395","id":103935355,"number":395,"title":"Removing CallSet as a first-class object; VCF merging and the N+1 problem","user":{"login":"mlin","id":356550,"avatar_url":"https://avatars.githubusercontent.com/u/356550?v=3","gravatar_id":"","url":"https://api.github.com/users/mlin","html_url":"https://github.com/mlin","followers_url":"https://api.github.com/users/mlin/followers","following_url":"https://api.github.com/users/mlin/following{/other_user}","gists_url":"https://api.github.com/users/mlin/gists{/gist_id}","starred_url":"https://api.github.com/users/mlin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlin/subscriptions","organizations_url":"https://api.github.com/users/mlin/orgs","repos_url":"https://api.github.com/users/mlin/repos","events_url":"https://api.github.com/users/mlin/events{/privacy}","received_events_url":"https://api.github.com/users/mlin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ga4gh/schemas/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-30T17:25:01Z","updated_at":"2015-08-30T17:58:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ga4gh/schemas/pulls/395","html_url":"https://github.com/ga4gh/schemas/pull/395","diff_url":"https://github.com/ga4gh/schemas/pull/395.diff","patch_url":"https://github.com/ga4gh/schemas/pull/395.patch"},"body":"(Background: @dglazer recently wrote a [nice recap](https://github.com/ga4gh/schemas/pull/391#issuecomment-134821662) of the conceptual model underlying the Reads+Variants API. This PR grew out of discussion in #383 and #391, and involves conceptual issues that have been aired previously #49 #73 #168, but perhaps not fully litigated in this group.)\r\n\r\nThe concept on trial in this PR is the CallSet, which represents a \"column of genotype calls\" in a VariantSet, roughly equivalent to one column of a multicolumnar VCF file. CallSets are first-class objects in our API, with their own 'created' and 'updated' timestamps, and optionally other metadata. This supports a mode of operations in which existing VariantSets can be merged together, such that the merged VariantSet composes several CallSets generated at different times and perhaps using different methods. Modeling this situation seems to be the main reason for CallSet to be a first-class object rather than constitutively a part of a single VariantSet.\r\n\r\nBecause our Variants schema adheres to the VCF data model, it inherits some fundamental difficulties with merging VCFs. First, there's distinguishing homozygous reference from no-call at variant sites not present in all CallSets. Second, it's difficult to construct a \"convenient\" representation of the merged VariantSet when there are multiple different overlapping variants in the CallSets. We'll elaborate on \"convenient\" below, but the result of these difficulties is that the generation of multicolumnar VariantSets is usually more complicated than the composition of CallSets contemplated above. There are often algorithmic steps involved that transform the inputs or supplement them with other data, such that the new VariantSet really consists of all-new CallSets generated together. This \"N+1 problem\" — the need to reprocess everything in some way when adding a new [analysis] sample to a study — is an outstanding challenge in the abstract VCF data model.\r\n\r\nNow that stated, simple compositional merging of existing VariantSets — e.g. joining VCF rows by exact (POS,REF,ALT) — can be a useful operation in practice. That type of procedure handles common cases like biallelic SNVs well, and VCF data can be prepared in particular ways that help to marginalize other problematic cases (e.g. reducing to allelic primitives, splitting multiallelic sites into overlapping biallelic sites, and including all homozygous reference genotypes). The residual issues/limitations with VCF merging may then be quite tolerable, depending on the use case.\r\n\r\n### The proposal\r\n\r\nThis pull request removes CallSet as a first-class API object. A column of calls becomes constitutively a part of one VariantSet and shares its metadata. API operations refer to columns by some \"handle\" string, unique at least within the VariantSet. This \"handle\" concept is a placeholder as other in-flight pull requests aim to ascribe additional meaning to them, upon which a different term such as \"analysis sample\" will be substituted.\r\n\r\n### The prosecution\r\n\r\n* Let's jettison CallSet because we don't need to provide a first-class abstraction mainly useful to model one rather simplistic way of merging VariantSets.\r\n* This doesn't *prevent* anyone from merging VariantSets however they like, creating new VariantSets with distinct columns.\r\n* It diminishes a potential illusion that the Global Alliance already has a sound general solution for VCF merging or the N+1 problem.\r\n* The CallSet concept has proven confusing to many of us, let alone outsiders, and removing it simplifies things. \r\n* We can add a CallSet-like object later, if/when we adopt a new model of genetic variation with easier merge semantics (which will probably require an effective rewrite of the Variants API anyway).\r\n\r\n### The defense\r\n\r\n* Let's keep our first-class CallSets because they're helpful to model VariantSet merging, a useful and commonly-practiced operation.\r\n* The corner cases where simple VariantSet merging causes problems are tolerable for many use cases, and it's much easier and more scalable than alternatives.\r\n* CallSet objects are more generally a useful place to stick any metadata applicable specifically to the (VariantSet, analysis sample) cross, regardless of how/when they were generated.\r\n* Just finding a better name for CallSet would suffice to reduce confusion.\r\n* Having first-class CallSets now will architecturally prepare implementations for future variation models with easier merge semantics.\r\n\r\n<hr/>\r\n\r\n### Appendix: how should a multicolumnar VariantSet look?\r\n\r\nBy \"convenient\", above, we mean a multicolumnar VariantSet would ideally present a \"squared-off\" matrix of genotypes, with non-overlapping variants, mutually-exclusive alleles, and the fewest possible missing entries. This is desirable for downstream analysts, because it's then straightforward to calculate population-genetic statistics, association tests, genotype posteriors, and partition functions. Those are much harder to formulate based on \"ragged\" genotypes, because of the need to meticulously account for the entanglement of overlapping variants and alleles. Indeed, beyond mere convenience, some propose that VCFs \"[SHOULD NOT](https://www.ietf.org/rfc/rfc2119.txt)\" contain overlapping variants [[1](https://github.com/ga4gh/schemas/issues/168#issuecomment-60795294),[2](https://github.com/ga4gh/schemas/issues/168#issuecomment-60789648)].\r\n\r\nCurrent formal methods for finding the squared-off genotype matrix are not really \"VCF merging\" *per se*, but rather probabilistic algorithms relying on lower-level forms of input data. The most challenging aspect of these algorithms is that the matrix is generated jointly, modeling all observations simultaneously in each row, and possibly across rows. Observing the N+1th sample sometimes causes revisions to all previous N samples!\r\n\r\nShort of formal joint genotyping, there are also *ad hoc* VCF merging procedures that work harder to achieve \"squareness\" than (POS,REF,ALT) joining, e.g. by unifying multiallelic sites and rewriting genotypes when possible, and using auxiliary read coverage information to adjudicate ref/no-call. Still, these are also generating *new* data rather than merely composing existing data.\r\n"},"comment":{"url":"https://api.github.com/repos/ga4gh/schemas/issues/comments/136164117","html_url":"https://github.com/ga4gh/schemas/pull/395#issuecomment-136164117","issue_url":"https://api.github.com/repos/ga4gh/schemas/issues/395","id":136164117,"user":{"login":"mlin","id":356550,"avatar_url":"https://avatars.githubusercontent.com/u/356550?v=3","gravatar_id":"","url":"https://api.github.com/users/mlin","html_url":"https://github.com/mlin","followers_url":"https://api.github.com/users/mlin/followers","following_url":"https://api.github.com/users/mlin/following{/other_user}","gists_url":"https://api.github.com/users/mlin/gists{/gist_id}","starred_url":"https://api.github.com/users/mlin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlin/subscriptions","organizations_url":"https://api.github.com/users/mlin/orgs","repos_url":"https://api.github.com/users/mlin/repos","events_url":"https://api.github.com/users/mlin/events{/privacy}","received_events_url":"https://api.github.com/users/mlin/received_events","type":"User","site_admin":false},"created_at":"2015-08-30T17:58:19Z","updated_at":"2015-08-30T17:58:19Z","body":"Since I opened this PR I guess I'm +1 but hopefully it's clear it's a close call to me, and I look forward to discussion. On the one hand I believe joint genotyping is a worthwhile best practice, and my group has a [side project](https://github.com/dnanexus-rnd/GLnexus/wiki) to scale it up. On the other hand we work with a number of customers with various *ad hoc* ways of merging VCFs and they're generally doing alright (the customer is always right!?)"}}