{"action":"created","issue":{"url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/1","labels_url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/1/comments","events_url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/1/events","html_url":"https://github.com/overview/overview-js-tokenizer/issues/1","id":115254031,"number":1,"title":"question: is it possible to also remove the non-token characters?","user":{"login":"retorquere","id":132108,"avatar_url":"https://avatars.githubusercontent.com/u/132108?v=3","gravatar_id":"","url":"https://api.github.com/users/retorquere","html_url":"https://github.com/retorquere","followers_url":"https://api.github.com/users/retorquere/followers","following_url":"https://api.github.com/users/retorquere/following{/other_user}","gists_url":"https://api.github.com/users/retorquere/gists{/gist_id}","starred_url":"https://api.github.com/users/retorquere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/retorquere/subscriptions","organizations_url":"https://api.github.com/users/retorquere/orgs","repos_url":"https://api.github.com/users/retorquere/repos","events_url":"https://api.github.com/users/retorquere/events{/privacy}","received_events_url":"https://api.github.com/users/retorquere/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-11-05T10:28:12Z","updated_at":"2015-12-04T15:40:32Z","closed_at":"2015-12-04T14:58:34Z","body":"So is there a way to get from `\"The cat's meowed 1,000 times! Really!\"` to `[ \"The\", \" \", \"cat's\", \" \" , \"meowed\", \" \", \"1,000\", \"!\", \"times\", \"Really\", \"!\" ]`"},"comment":{"url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/comments/161999117","html_url":"https://github.com/overview/overview-js-tokenizer/issues/1#issuecomment-161999117","issue_url":"https://api.github.com/repos/overview/overview-js-tokenizer/issues/1","id":161999117,"user":{"login":"adamhooper","id":10812,"avatar_url":"https://avatars.githubusercontent.com/u/10812?v=3","gravatar_id":"","url":"https://api.github.com/users/adamhooper","html_url":"https://github.com/adamhooper","followers_url":"https://api.github.com/users/adamhooper/followers","following_url":"https://api.github.com/users/adamhooper/following{/other_user}","gists_url":"https://api.github.com/users/adamhooper/gists{/gist_id}","starred_url":"https://api.github.com/users/adamhooper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamhooper/subscriptions","organizations_url":"https://api.github.com/users/adamhooper/orgs","repos_url":"https://api.github.com/users/adamhooper/repos","events_url":"https://api.github.com/users/adamhooper/events{/privacy}","received_events_url":"https://api.github.com/users/adamhooper/received_events","type":"User","site_admin":false},"created_at":"2015-12-04T15:40:32Z","updated_at":"2015-12-04T15:40:32Z","body":"https://github.com/overview/overview-js-tokenizer/commits/master"}}