{"action":"opened","issue":{"url":"https://api.github.com/repos/Ulflander/compendium-js/issues/3","labels_url":"https://api.github.com/repos/Ulflander/compendium-js/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/Ulflander/compendium-js/issues/3/comments","events_url":"https://api.github.com/repos/Ulflander/compendium-js/issues/3/events","html_url":"https://github.com/Ulflander/compendium-js/issues/3","id":92803886,"number":3,"title":"Extra spaces added to raw property","user":{"login":"dhrubins","id":5916119,"avatar_url":"https://avatars.githubusercontent.com/u/5916119?v=3","gravatar_id":"","url":"https://api.github.com/users/dhrubins","html_url":"https://github.com/dhrubins","followers_url":"https://api.github.com/users/dhrubins/followers","following_url":"https://api.github.com/users/dhrubins/following{/other_user}","gists_url":"https://api.github.com/users/dhrubins/gists{/gist_id}","starred_url":"https://api.github.com/users/dhrubins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhrubins/subscriptions","organizations_url":"https://api.github.com/users/dhrubins/orgs","repos_url":"https://api.github.com/users/dhrubins/repos","events_url":"https://api.github.com/users/dhrubins/events{/privacy}","received_events_url":"https://api.github.com/users/dhrubins/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-03T06:30:29Z","updated_at":"2015-07-03T06:30:29Z","closed_at":null,"body":"Hey Xav,\r\nIn the object that gets returned, the raw text seems to have some extra spaces between tokens. For example, I would expect that running compendium.analyse('My name is Dr. Jekyll.'); would return the original text as the 'raw' property, as follows:\r\n\r\n\t[ { time: 9,                        // Time of processing, in ms\r\n\t    length: 6,                      // Count of tokens\r\n\t    raw: 'My name is Dr. Jekyll.', // Raw string\r\n\t    stats: ...\r\n\r\nHowever, it actually returns the following:\r\n\r\n\t[ { time: 9,                        // Time of processing, in ms\r\n\t    length: 6,                      // Count of tokens\r\n\t    raw: 'My name is Dr. Jekyll .', // Raw string\r\n\t    stats: ...\r\n\r\nIt's a bit more pronounced with extra punctuation, since those are tokenized separately:\r\n\r\ncompendium.analyse('Today is 4/2/2015, or 2/4/2015- depending on where in the world you live!');\r\n\r\n\t[ { time: 6,                        // Time of processing, in ms\r\n\t    length: 23,                     // Count of tokens\r\n\t    raw: 'Today is 4 / 2 / 2015 , or 2 / 4 / 2015- depending on where in the world you live !', // Raw string\r\n\t    stats: ...\r\n\r\nIt's a minor issue, but it can cause some presentation weirdness.\r\n\r\nA quick look at the code makes me think it's happening on line 41 of detector.s.1.entities.js but that's only a first glance.\r\n\r\nBTW, great work on this package!"}}