{"action":"closed","issue":{"url":"https://api.github.com/repos/Rostlab/nalaf/issues/157","labels_url":"https://api.github.com/repos/Rostlab/nalaf/issues/157/labels{/name}","comments_url":"https://api.github.com/repos/Rostlab/nalaf/issues/157/comments","events_url":"https://api.github.com/repos/Rostlab/nalaf/issues/157/events","html_url":"https://github.com/Rostlab/nalaf/issues/157","id":120744348,"number":157,"title":"Tokenizer non- ascii chars","user":{"login":"abojchevski","id":8150330,"avatar_url":"https://avatars.githubusercontent.com/u/8150330?v=3","gravatar_id":"","url":"https://api.github.com/users/abojchevski","html_url":"https://github.com/abojchevski","followers_url":"https://api.github.com/users/abojchevski/followers","following_url":"https://api.github.com/users/abojchevski/following{/other_user}","gists_url":"https://api.github.com/users/abojchevski/gists{/gist_id}","starred_url":"https://api.github.com/users/abojchevski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abojchevski/subscriptions","organizations_url":"https://api.github.com/users/abojchevski/orgs","repos_url":"https://api.github.com/users/abojchevski/repos","events_url":"https://api.github.com/users/abojchevski/events{/privacy}","received_events_url":"https://api.github.com/users/abojchevski/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"abojchevski","id":8150330,"avatar_url":"https://avatars.githubusercontent.com/u/8150330?v=3","gravatar_id":"","url":"https://api.github.com/users/abojchevski","html_url":"https://github.com/abojchevski","followers_url":"https://api.github.com/users/abojchevski/followers","following_url":"https://api.github.com/users/abojchevski/following{/other_user}","gists_url":"https://api.github.com/users/abojchevski/gists{/gist_id}","starred_url":"https://api.github.com/users/abojchevski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abojchevski/subscriptions","organizations_url":"https://api.github.com/users/abojchevski/orgs","repos_url":"https://api.github.com/users/abojchevski/repos","events_url":"https://api.github.com/users/abojchevski/events{/privacy}","received_events_url":"https://api.github.com/users/abojchevski/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Rostlab/nalaf/milestones/6","html_url":"https://github.com/Rostlab/nalaf/milestones/2-ThisWeek","labels_url":"https://api.github.com/repos/Rostlab/nalaf/milestones/6/labels","id":1109896,"number":6,"title":"2-ThisWeek","description":"","creator":{"login":"carstenuhlig","id":1842645,"avatar_url":"https://avatars.githubusercontent.com/u/1842645?v=3","gravatar_id":"","url":"https://api.github.com/users/carstenuhlig","html_url":"https://github.com/carstenuhlig","followers_url":"https://api.github.com/users/carstenuhlig/followers","following_url":"https://api.github.com/users/carstenuhlig/following{/other_user}","gists_url":"https://api.github.com/users/carstenuhlig/gists{/gist_id}","starred_url":"https://api.github.com/users/carstenuhlig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/carstenuhlig/subscriptions","organizations_url":"https://api.github.com/users/carstenuhlig/orgs","repos_url":"https://api.github.com/users/carstenuhlig/repos","events_url":"https://api.github.com/users/carstenuhlig/events{/privacy}","received_events_url":"https://api.github.com/users/carstenuhlig/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":2,"state":"open","created_at":"2015-05-14T09:09:12Z","updated_at":"2015-12-22T08:02:06Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-12-07T10:39:47Z","updated_at":"2015-12-22T08:02:06Z","closed_at":"2015-12-22T08:02:06Z","body":"From looking at the learned state features by the CRF model, it appreas that some non-ascii characters might have a big impact (for example Δ). Our current tokenizer does not separate them into separate tokens so the model ends up learning bunch of over-fitted specific weights.\r\nTherefore:\r\n* [x] Separate non-ascii chars into their own tokens\r\n* [x] Compare performance"}}