{"action":"created","issue":{"url":"https://api.github.com/repos/nlplab/brat/issues/1115","labels_url":"https://api.github.com/repos/nlplab/brat/issues/1115/labels{/name}","comments_url":"https://api.github.com/repos/nlplab/brat/issues/1115/comments","events_url":"https://api.github.com/repos/nlplab/brat/issues/1115/events","html_url":"https://github.com/nlplab/brat/issues/1115","id":45331864,"number":1115,"title":"Special characters display problem","user":{"login":"nguyend","id":6589663,"avatar_url":"https://avatars.githubusercontent.com/u/6589663?v=3","gravatar_id":"","url":"https://api.github.com/users/nguyend","html_url":"https://github.com/nguyend","followers_url":"https://api.github.com/users/nguyend/followers","following_url":"https://api.github.com/users/nguyend/following{/other_user}","gists_url":"https://api.github.com/users/nguyend/gists{/gist_id}","starred_url":"https://api.github.com/users/nguyend/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nguyend/subscriptions","organizations_url":"https://api.github.com/users/nguyend/orgs","repos_url":"https://api.github.com/users/nguyend/repos","events_url":"https://api.github.com/users/nguyend/events{/privacy}","received_events_url":"https://api.github.com/users/nguyend/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/nlplab/brat/labels/req-from-user","name":"req-from-user","color":"c8ff75"}],"state":"open","locked":false,"assignee":{"login":"spyysalo","id":587850,"avatar_url":"https://avatars.githubusercontent.com/u/587850?v=3","gravatar_id":"","url":"https://api.github.com/users/spyysalo","html_url":"https://github.com/spyysalo","followers_url":"https://api.github.com/users/spyysalo/followers","following_url":"https://api.github.com/users/spyysalo/following{/other_user}","gists_url":"https://api.github.com/users/spyysalo/gists{/gist_id}","starred_url":"https://api.github.com/users/spyysalo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spyysalo/subscriptions","organizations_url":"https://api.github.com/users/spyysalo/orgs","repos_url":"https://api.github.com/users/spyysalo/repos","events_url":"https://api.github.com/users/spyysalo/events{/privacy}","received_events_url":"https://api.github.com/users/spyysalo/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/nlplab/brat/milestones/19","html_url":"https://github.com/nlplab/brat/milestones/v1.5%20Amaze-a-Vole","labels_url":"https://api.github.com/repos/nlplab/brat/milestones/19/labels","id":198568,"number":19,"title":"v1.5 Amaze-a-Vole","description":"Naming from one of the incorporated company names at the end of the sketch [Confuse-a-Cat](http://www.youtube.com/watch?v=B2Je1CEPkUM).","creator":{"login":"ninjin","id":354934,"avatar_url":"https://avatars.githubusercontent.com/u/354934?v=3","gravatar_id":"","url":"https://api.github.com/users/ninjin","html_url":"https://github.com/ninjin","followers_url":"https://api.github.com/users/ninjin/followers","following_url":"https://api.github.com/users/ninjin/following{/other_user}","gists_url":"https://api.github.com/users/ninjin/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjin/subscriptions","organizations_url":"https://api.github.com/users/ninjin/orgs","repos_url":"https://api.github.com/users/ninjin/repos","events_url":"https://api.github.com/users/ninjin/events{/privacy}","received_events_url":"https://api.github.com/users/ninjin/received_events","type":"User","site_admin":false},"open_issues":259,"closed_issues":5,"state":"open","created_at":"2012-10-20T06:12:20Z","updated_at":"2015-07-26T18:32:18Z","due_on":"2015-05-01T07:00:00Z","closed_at":null},"comments":5,"created_at":"2014-10-09T06:21:11Z","updated_at":"2015-08-07T04:43:18Z","closed_at":null,"body":"Hi, \r\nI tried to annotate some tweets but it seems the client did not displayed the \"emotion\" character well.\r\nAll the spaces are missing after this 😂 (attached snapshot).\r\nAnd the page is not responding with these 01F44A, 01F4AA,01F48B\r\n\r\nPlease suggest how to deal with these character either remove or keep it.\r\nThanks and Regards,\r\n\r\n![image](https://cloud.githubusercontent.com/assets/6589663/4571717/3be038cc-4f7c-11e4-96b1-ee4059cdea30.png)\r\n"},"comment":{"url":"https://api.github.com/repos/nlplab/brat/issues/comments/128594508","html_url":"https://github.com/nlplab/brat/issues/1115#issuecomment-128594508","issue_url":"https://api.github.com/repos/nlplab/brat/issues/1115","id":128594508,"user":{"login":"amadanmath","id":134244,"avatar_url":"https://avatars.githubusercontent.com/u/134244?v=3","gravatar_id":"","url":"https://api.github.com/users/amadanmath","html_url":"https://github.com/amadanmath","followers_url":"https://api.github.com/users/amadanmath/followers","following_url":"https://api.github.com/users/amadanmath/following{/other_user}","gists_url":"https://api.github.com/users/amadanmath/gists{/gist_id}","starred_url":"https://api.github.com/users/amadanmath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amadanmath/subscriptions","organizations_url":"https://api.github.com/users/amadanmath/orgs","repos_url":"https://api.github.com/users/amadanmath/repos","events_url":"https://api.github.com/users/amadanmath/events{/privacy}","received_events_url":"https://api.github.com/users/amadanmath/received_events","type":"User","site_admin":false},"created_at":"2015-08-07T04:43:18Z","updated_at":"2015-08-07T04:43:18Z","body":"It seems that the display bug has already been fixed in master branch, as I can't reproduce it.\r\n\r\nHowever, there is a more insidious bug here: all annotations for text that follows any astral characters will be wrong (offset by one for each preceding astral character). The fact that it even works is an amazing coincidence owing to the fact that Python2 and JavaScript both don't know how to handle astral characters, and break in the exactly same way: by treating them as if they were two characters. Thus, if you annotate `\"yes\"` in `\"😂😂 yes\"`, the standoff should be 3-6 (the first three characters being emoji, emoji and space); but it is recorded as 5-8 (the first five characters being emoji-part-1, emoji-part-2, emoji-part-1, emoji-part-2 and space).\r\n\r\nThere are three ways around it that I can see, and none are pretty:\r\n\r\n* Fix annotations by postprocessing (deduce 1 from each standoff index for each astral character before it)\r\n* Replace all astral characters by basic plane stand-ins\r\n* Replace any use of `length` property, `substr`, `substring`, `charAt` and `charCodeAt` methods (and Python equivalents) on strings with unicode-compliant versions (huge task, I'm very unlikely to do it).\r\n\r\nNow that I think about it, this should probably be a separate issue..."}}