{"action":"opened","issue":{"url":"https://api.github.com/repos/jpsember/tokn/issues/3","labels_url":"https://api.github.com/repos/jpsember/tokn/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/jpsember/tokn/issues/3/comments","events_url":"https://api.github.com/repos/jpsember/tokn/issues/3/events","html_url":"https://github.com/jpsember/tokn/issues/3","id":89861523,"number":3,"title":"Tokenization is not occurring correctly","user":{"login":"jpsember","id":4098788,"avatar_url":"https://avatars.githubusercontent.com/u/4098788?v=3","gravatar_id":"","url":"https://api.github.com/users/jpsember","html_url":"https://github.com/jpsember","followers_url":"https://api.github.com/users/jpsember/followers","following_url":"https://api.github.com/users/jpsember/following{/other_user}","gists_url":"https://api.github.com/users/jpsember/gists{/gist_id}","starred_url":"https://api.github.com/users/jpsember/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpsember/subscriptions","organizations_url":"https://api.github.com/users/jpsember/orgs","repos_url":"https://api.github.com/users/jpsember/repos","events_url":"https://api.github.com/users/jpsember/events{/privacy}","received_events_url":"https://api.github.com/users/jpsember/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-21T04:13:14Z","updated_at":"2015-06-21T04:13:14Z","closed_at":null,"body":"For example, with the following token definitions:\r\n\r\n```\r\nWS: \\s+\r\nA: a\r\n```\r\n\r\nwhen we attempt to tokenize this string:\r\n```\r\n' a  aaa a'\r\n```\r\n\r\nwe get:\r\n```\r\n read 1: ' a'\r\n read 1: '  a'\r\n read 1: 'a'\r\n read 1: 'a'\r\n read 1: ' a'\r\n```\r\n"}}