{"action":"created","issue":{"url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/3","labels_url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/3/comments","events_url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/3/events","html_url":"https://github.com/ufal/lindat-corpora-conversions/issues/3","id":102789807,"number":3,"title":"UD - fused tokens","user":{"login":"natalink","id":2538188,"avatar_url":"https://avatars.githubusercontent.com/u/2538188?v=3","gravatar_id":"","url":"https://api.github.com/users/natalink","html_url":"https://github.com/natalink","followers_url":"https://api.github.com/users/natalink/followers","following_url":"https://api.github.com/users/natalink/following{/other_user}","gists_url":"https://api.github.com/users/natalink/gists{/gist_id}","starred_url":"https://api.github.com/users/natalink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/natalink/subscriptions","organizations_url":"https://api.github.com/users/natalink/orgs","repos_url":"https://api.github.com/users/natalink/repos","events_url":"https://api.github.com/users/natalink/events{/privacy}","received_events_url":"https://api.github.com/users/natalink/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/labels/Universal%20dependencies","name":"Universal dependencies","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-24T13:10:38Z","updated_at":"2015-08-31T13:17:55Z","closed_at":null,"body":"The following treebanks have fused tokens:  de,it,cs,es,fa,fi,fr,he, e.g. 2-3 and 5-6 in Spanish, example in conllu:\r\n```\r\n1       Producto        _       NOUN    _       _       15      nmod    _       _\r\n2-3     del     _       _       _       _       _       _       _       _\r\n2       de      _       ADP     _       _       4       case    _       _\r\n3       el      _       DET     _       _       4       det     _       _\r\n4       fin     _       NOUN    _       _       1       nmod    _       _\r\n5-6     del     _       _       _       _       _       _       _       _\r\n5       de      _       ADP     _       _       7       case    _       _\r\n6       el      _       DET     _       _       7       det     _       _\r\n7       imperio _       NOUN    _       _       4       nmod    _       _\r\n8       y       _       CONJ    _       _       7       cc      _       _\r\n9       las     _       DET     _       _       10      det     _       _\r\n10      invasiones      _       NOUN    _       _       7       conj    _\r\n```\r\nThere are several ways to handle this problem, as suggested by Dan Zeman:\r\n\r\n* **Postprocess treex output. Ignore 2-3 lines.** If we leave them as they are, the nodes 2-3 containing the actual word form will be ignored by treex. We can put the wordform *del* into the position of wordform *2 de* and concatenate other features, e.g. *ADP+DET*. Lemma can be either a wordform or *de+el*. The wordform in line 3 will be empty to preserve original text:\r\n```\r\ndel     del     ADP+DET _       case+det        fin     _+_     NOUN    _       nmod    right   distant\r\n        _       DET     _       det     fin     _       NOUN    _       nmod    right   immediate\r\nfin     _       NOUN    _       nmod    Producto        _       NOUN    _       nmod    left    distant\r\n```\r\nTwo major drawbacks. It will leave a phantom node with no wordform and other attributes. In the above example, *de* and *el* have a parent *fin*, but the third not is immediate to the parent whereas\r\nthe second is distant.\r\n\r\n* **Preprocess before treex** (process_fusion.py in data/treex/universal_dep/scripts). Merge the lines 2 and 3 into one line, shift the ids of subsequent tokens:\r\n```\r\n1       Producto        _       NOUN    _       _       15      nmod    _       _\r\n2     del     del       ADP+DET       _+_       _+_       4       case+det       _+_       _+_\r\n3       fin     _       NOUN    _       _       1       nmod    _       _\r\n```\r\nThe problem with distant and position or phantom nodes will not occur. So far the algorithm\r\ndoes not do the shift of heads whereas doing the shift of ids. If this solution is acceptable,\r\nI will fix it.\r\n\r\n* Think of some more complicated way to handle the problem so that all information from UD will\r\nbe in KonText.\r\n"},"comment":{"url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/comments/136368479","html_url":"https://github.com/ufal/lindat-corpora-conversions/issues/3#issuecomment-136368479","issue_url":"https://api.github.com/repos/ufal/lindat-corpora-conversions/issues/3","id":136368479,"user":{"login":"dan-zeman","id":663752,"avatar_url":"https://avatars.githubusercontent.com/u/663752?v=3","gravatar_id":"","url":"https://api.github.com/users/dan-zeman","html_url":"https://github.com/dan-zeman","followers_url":"https://api.github.com/users/dan-zeman/followers","following_url":"https://api.github.com/users/dan-zeman/following{/other_user}","gists_url":"https://api.github.com/users/dan-zeman/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-zeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-zeman/subscriptions","organizations_url":"https://api.github.com/users/dan-zeman/orgs","repos_url":"https://api.github.com/users/dan-zeman/repos","events_url":"https://api.github.com/users/dan-zeman/events{/privacy}","received_events_url":"https://api.github.com/users/dan-zeman/received_events","type":"User","site_admin":false},"created_at":"2015-08-31T13:17:55Z","updated_at":"2015-08-31T13:17:55Z","body":"I think we need a better representation of fused tokens in Treex. Now it is just sketched using the wild attributes but it will probably be needed in future, as it is part of the UD guidelines. So we need a less wild solution. Once we have it, we could try to implement directly in Treex the heuristics that will collapse fused words whenever desirable. And once we have this, we should probably use it before exporting data for Kontext. Because the surface matters here."}}