{"action":"created","issue":{"url":"https://api.github.com/repos/UniversalDependencies/docs/issues/153","labels_url":"https://api.github.com/repos/UniversalDependencies/docs/issues/153/labels{/name}","comments_url":"https://api.github.com/repos/UniversalDependencies/docs/issues/153/comments","events_url":"https://api.github.com/repos/UniversalDependencies/docs/issues/153/events","html_url":"https://github.com/UniversalDependencies/docs/issues/153","id":60648523,"number":153,"title":"Unclear how to reconcile token and sub-token annotation","user":{"login":"reckart","id":1410238,"avatar_url":"https://avatars.githubusercontent.com/u/1410238?v=3","gravatar_id":"","url":"https://api.github.com/users/reckart","html_url":"https://github.com/reckart","followers_url":"https://api.github.com/users/reckart/followers","following_url":"https://api.github.com/users/reckart/following{/other_user}","gists_url":"https://api.github.com/users/reckart/gists{/gist_id}","starred_url":"https://api.github.com/users/reckart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reckart/subscriptions","organizations_url":"https://api.github.com/users/reckart/orgs","repos_url":"https://api.github.com/users/reckart/repos","events_url":"https://api.github.com/users/reckart/events{/privacy}","received_events_url":"https://api.github.com/users/reckart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-03-11T12:04:47Z","updated_at":"2015-03-11T15:03:47Z","closed_at":null,"body":"I have looked into implementing a reader and writer for CoNLL-U for DKPro Core, but I have some doubts regarding the format specification. The examples below come from the CoNLL-U format specification and documentation.\r\n\r\n*Example 1*\r\n\r\n    3-4    al        _\r\n    3      a         a\r\n    4      el        el\r\n\r\nI assume this snippet means that the document text contains \"al\" and not \"a el\" - so when reading this, \"al\" is what I would like to interpret as document text. So \"a\" and \"el\" would become annotations on the \"al\" text span. However, as it is, it is impossible relate the subtokens to offsets in the main token, e.g. information that \"el\" corresponds to the \"l\" in \"al\". So I suppose I can only ignore the subtokens. But then hits the next problem that most annotations are not available on the main token, e.g.:\r\n\r\n*Example 2*\r\n\r\n    2-3   haven't   _      _       _     _                                 _   _\r\n    2     have      have   VERB    VB    Number=Sing|Person=1|Tense=Pres   0   root\r\n    3     not       not    PART    RB    Negative=Neg                      2   neg\r\n\r\nThe documentation says that it is easy to extract an approximation like this:\r\n\r\n*Example 3*\r\n\r\n2     haven't   _      VERB    _     Negative=Neg|Number=Sing|Person=1|Tense=Pres   0   root\r\n\r\nbut it does not specify any annotation-independent rules on how to do that... and it loses information. I believe format alone does not contain enough information to condense Example 2 to Example 3.\r\n\r\nSo I suppose my questions are: \r\n* are there any (tagset-independent) rules to aggregate sub-token annotations to the token-level?\r\n* would it be feasible to extend the format to allow sub-tokens to refer to offsets within the main token, e.g.\r\n\r\n```\r\n2-3   haven't   _      _       _     _                                 _   _\r\n2(0-4)     have      have   VERB    VB    Number=Sing|Person=1|Tense=Pres   0   root\r\n3(4-7)     not       not    PART    RB    Negative=Neg                      2   neg\r\n```"},"comment":{"url":"https://api.github.com/repos/UniversalDependencies/docs/issues/comments/78280851","html_url":"https://github.com/UniversalDependencies/docs/issues/153#issuecomment-78280851","issue_url":"https://api.github.com/repos/UniversalDependencies/docs/issues/153","id":78280851,"user":{"login":"jnivre","id":7601366,"avatar_url":"https://avatars.githubusercontent.com/u/7601366?v=3","gravatar_id":"","url":"https://api.github.com/users/jnivre","html_url":"https://github.com/jnivre","followers_url":"https://api.github.com/users/jnivre/followers","following_url":"https://api.github.com/users/jnivre/following{/other_user}","gists_url":"https://api.github.com/users/jnivre/gists{/gist_id}","starred_url":"https://api.github.com/users/jnivre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnivre/subscriptions","organizations_url":"https://api.github.com/users/jnivre/orgs","repos_url":"https://api.github.com/users/jnivre/repos","events_url":"https://api.github.com/users/jnivre/events{/privacy}","received_events_url":"https://api.github.com/users/jnivre/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T15:03:47Z","updated_at":"2015-03-11T15:03:47Z","body":"I implemented a heuristic conversion in the first validation script. I am not sure what Filip has done with this after he took charge of the validator, but it should be available somewhere. There can definitely be cases where the two subwords of a token do not have the same parent, so the conversion will always be heuristic. Character offsets are not used in UD and wouldn't help with the conversion anyway as far as I can see. :)"}}