{"action":"created","issue":{"url":"https://api.github.com/repos/iit-cs579/main/issues/106","labels_url":"https://api.github.com/repos/iit-cs579/main/issues/106/labels{/name}","comments_url":"https://api.github.com/repos/iit-cs579/main/issues/106/comments","events_url":"https://api.github.com/repos/iit-cs579/main/issues/106/events","html_url":"https://github.com/iit-cs579/main/issues/106","id":115122409,"number":106,"title":"slightly high values of accuracy in tokenizer_expt..fine or bug?","user":{"login":"hinagarg","id":10472323,"avatar_url":"https://avatars.githubusercontent.com/u/10472323?v=3","gravatar_id":"","url":"https://api.github.com/users/hinagarg","html_url":"https://github.com/hinagarg","followers_url":"https://api.github.com/users/hinagarg/followers","following_url":"https://api.github.com/users/hinagarg/following{/other_user}","gists_url":"https://api.github.com/users/hinagarg/gists{/gist_id}","starred_url":"https://api.github.com/users/hinagarg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hinagarg/subscriptions","organizations_url":"https://api.github.com/users/hinagarg/orgs","repos_url":"https://api.github.com/users/hinagarg/repos","events_url":"https://api.github.com/users/hinagarg/events{/privacy}","received_events_url":"https://api.github.com/users/hinagarg/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-04T18:52:14Z","updated_at":"2015-11-07T03:50:05Z","closed_at":"2015-11-05T00:22:43Z","body":"Hi,\r\n\r\nExpected values of accuracy in tokenizer_expt method is:\r\n\r\n[0.73999999999999999, 0.74499999999999988, 0.74749999999999994]\r\n\r\nBut my output is:\r\n\r\n[0.73999999999999999, 0.75500000000000012, 0.75749999999999995]\r\n\r\nIs my output fine?"},"comment":{"url":"https://api.github.com/repos/iit-cs579/main/issues/comments/154614394","html_url":"https://github.com/iit-cs579/main/issues/106#issuecomment-154614394","issue_url":"https://api.github.com/repos/iit-cs579/main/issues/106","id":154614394,"user":{"login":"hinagarg","id":10472323,"avatar_url":"https://avatars.githubusercontent.com/u/10472323?v=3","gravatar_id":"","url":"https://api.github.com/users/hinagarg","html_url":"https://github.com/hinagarg","followers_url":"https://api.github.com/users/hinagarg/followers","following_url":"https://api.github.com/users/hinagarg/following{/other_user}","gists_url":"https://api.github.com/users/hinagarg/gists{/gist_id}","starred_url":"https://api.github.com/users/hinagarg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hinagarg/subscriptions","organizations_url":"https://api.github.com/users/hinagarg/orgs","repos_url":"https://api.github.com/users/hinagarg/repos","events_url":"https://api.github.com/users/hinagarg/events{/privacy}","received_events_url":"https://api.github.com/users/hinagarg/received_events","type":"User","site_admin":false},"created_at":"2015-11-07T03:50:05Z","updated_at":"2015-11-07T03:50:05Z","body":"Yea, the problem was in tokenizer_punct .. try to use other examples given by professor as below:\r\n\r\nnot the fiancée of not Henri\r\ntokenize: not, the, fianc, e, of, not, henri\r\ntokenize_with_punct: not, the, fianc, é, e, of, not, henri\r\ntokenize_with_not: not, not_the, not_fianc, é, e, of, not, not_henri\r\n\r\n\r\ntokenize_with_punct(u'hi there\\n how are you?')\r\n[u'hi', u'there', u'how', u'are', u'you', u'?']\r\ntokenize_with_punct(u'hi there \\x85 how are you?')\r\n[u'hi', u'there', u'how', u'are', u'you', u'?']\r\ntokenize_with_punct(u'hi there \\r how are you?')\r\n[u'hi', u'there', u'how', u'are', u'you', u'?']\r\n\r\nfix ur tokenizer_punct ..u will get correct values"}}