{"action":"created","comment":{"url":"https://api.github.com/repos/haifengl/smile/pulls/comments/28297358","id":28297358,"diff_hunk":"@@ -0,0 +1,18 @@\n+package smile.util;\n+\n+import com.google.common.base.Splitter;\n+import com.google.common.collect.Sets;\n+\n+import java.util.Set;\n+\n+/**\n+ * @author Qiyang Zuo\n+ * @since 15/4/9\n+ */\n+public class Tokenizer {","path":"Smile/src/main/java/smile/util/Tokenizer.java","position":12,"original_position":12,"commit_id":"f9ec487a7b8316e1de22ffc038ba459e26ba4676","original_commit_id":"f9ec487a7b8316e1de22ffc038ba459e26ba4676","user":{"login":"haifengl","id":5502052,"avatar_url":"https://avatars.githubusercontent.com/u/5502052?v=3","gravatar_id":"","url":"https://api.github.com/users/haifengl","html_url":"https://github.com/haifengl","followers_url":"https://api.github.com/users/haifengl/followers","following_url":"https://api.github.com/users/haifengl/following{/other_user}","gists_url":"https://api.github.com/users/haifengl/gists{/gist_id}","starred_url":"https://api.github.com/users/haifengl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haifengl/subscriptions","organizations_url":"https://api.github.com/users/haifengl/orgs","repos_url":"https://api.github.com/users/haifengl/repos","events_url":"https://api.github.com/users/haifengl/events{/privacy}","received_events_url":"https://api.github.com/users/haifengl/received_events","type":"User","site_admin":false},"body":"We have tokenizer in SmileNLP.","created_at":"2015-04-14T02:46:26Z","updated_at":"2015-04-14T02:46:26Z","html_url":"https://github.com/haifengl/smile/pull/24#discussion_r28297358","pull_request_url":"https://api.github.com/repos/haifengl/smile/pulls/24","_links":{"self":{"href":"https://api.github.com/repos/haifengl/smile/pulls/comments/28297358"},"html":{"href":"https://github.com/haifengl/smile/pull/24#discussion_r28297358"},"pull_request":{"href":"https://api.github.com/repos/haifengl/smile/pulls/24"}}},"pull_request":{"url":"https://api.github.com/repos/haifengl/smile/pulls/24","id":32419115,"html_url":"https://github.com/haifengl/smile/pull/24","diff_url":"https://github.com/haifengl/smile/pull/24.diff","patch_url":"https://github.com/haifengl/smile/pull/24.patch","issue_url":"https://api.github.com/repos/haifengl/smile/issues/24","number":24,"state":"open","locked":false,"title":"add simhash","user":{"login":"kid1412z","id":1208327,"avatar_url":"https://avatars.githubusercontent.com/u/1208327?v=3","gravatar_id":"","url":"https://api.github.com/users/kid1412z","html_url":"https://github.com/kid1412z","followers_url":"https://api.github.com/users/kid1412z/followers","following_url":"https://api.github.com/users/kid1412z/following{/other_user}","gists_url":"https://api.github.com/users/kid1412z/gists{/gist_id}","starred_url":"https://api.github.com/users/kid1412z/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kid1412z/subscriptions","organizations_url":"https://api.github.com/users/kid1412z/orgs","repos_url":"https://api.github.com/users/kid1412z/repos","events_url":"https://api.github.com/users/kid1412z/events{/privacy}","received_events_url":"https://api.github.com/users/kid1412z/received_events","type":"User","site_admin":false},"body":"Hi Haifeng,\r\n    I add the LSH search based on signature of simhash, during the implementation, I found that there are a lot of repeated code and we cannot reuse some code already has been implemented. I think we may extract the Hash functions as *devices* so that all LSH search could share them. But it's not quite easy when I tried, because even the Euclidean hash functions used by LSH and MPLSH is slightly different from each other. It will take time to do this refactor.\r\n    BTW, do you have some recommended data sets to test this commit?\r\n### commit summary:\r\n* add simhash(simply use weights[1,1,1...1]) and the LSH search for signatures","created_at":"2015-04-01T13:35:35Z","updated_at":"2015-04-14T02:46:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7435bfdc6443997a9362b27fd909fa22a0f064a9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/haifengl/smile/pulls/24/commits","review_comments_url":"https://api.github.com/repos/haifengl/smile/pulls/24/comments","review_comment_url":"https://api.github.com/repos/haifengl/smile/pulls/comments{/number}","comments_url":"https://api.github.com/repos/haifengl/smile/issues/24/comments","statuses_url":"https://api.github.com/repos/haifengl/smile/statuses/f9ec487a7b8316e1de22ffc038ba459e26ba4676","head":{"label":"kid1412z:master","ref":"master","sha":"f9ec487a7b8316e1de22ffc038ba459e26ba4676","user":{"login":"kid1412z","id":1208327,"avatar_url":"https://avatars.githubusercontent.com/u/1208327?v=3","gravatar_id":"","url":"https://api.github.com/users/kid1412z","html_url":"https://github.com/kid1412z","followers_url":"https://api.github.com/users/kid1412z/followers","following_url":"https://api.github.com/users/kid1412z/following{/other_user}","gists_url":"https://api.github.com/users/kid1412z/gists{/gist_id}","starred_url":"https://api.github.com/users/kid1412z/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kid1412z/subscriptions","organizations_url":"https://api.github.com/users/kid1412z/orgs","repos_url":"https://api.github.com/users/kid1412z/repos","events_url":"https://api.github.com/users/kid1412z/events{/privacy}","received_events_url":"https://api.github.com/users/kid1412z/received_events","type":"User","site_admin":false},"repo":{"id":32738287,"name":"smile","full_name":"kid1412z/smile","owner":{"login":"kid1412z","id":1208327,"avatar_url":"https://avatars.githubusercontent.com/u/1208327?v=3","gravatar_id":"","url":"https://api.github.com/users/kid1412z","html_url":"https://github.com/kid1412z","followers_url":"https://api.github.com/users/kid1412z/followers","following_url":"https://api.github.com/users/kid1412z/following{/other_user}","gists_url":"https://api.github.com/users/kid1412z/gists{/gist_id}","starred_url":"https://api.github.com/users/kid1412z/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kid1412z/subscriptions","organizations_url":"https://api.github.com/users/kid1412z/orgs","repos_url":"https://api.github.com/users/kid1412z/repos","events_url":"https://api.github.com/users/kid1412z/events{/privacy}","received_events_url":"https://api.github.com/users/kid1412z/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kid1412z/smile","description":"Statistical Machine Intelligence & Learning Engine","fork":true,"url":"https://api.github.com/repos/kid1412z/smile","forks_url":"https://api.github.com/repos/kid1412z/smile/forks","keys_url":"https://api.github.com/repos/kid1412z/smile/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kid1412z/smile/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kid1412z/smile/teams","hooks_url":"https://api.github.com/repos/kid1412z/smile/hooks","issue_events_url":"https://api.github.com/repos/kid1412z/smile/issues/events{/number}","events_url":"https://api.github.com/repos/kid1412z/smile/events","assignees_url":"https://api.github.com/repos/kid1412z/smile/assignees{/user}","branches_url":"https://api.github.com/repos/kid1412z/smile/branches{/branch}","tags_url":"https://api.github.com/repos/kid1412z/smile/tags","blobs_url":"https://api.github.com/repos/kid1412z/smile/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kid1412z/smile/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kid1412z/smile/git/refs{/sha}","trees_url":"https://api.github.com/repos/kid1412z/smile/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kid1412z/smile/statuses/{sha}","languages_url":"https://api.github.com/repos/kid1412z/smile/languages","stargazers_url":"https://api.github.com/repos/kid1412z/smile/stargazers","contributors_url":"https://api.github.com/repos/kid1412z/smile/contributors","subscribers_url":"https://api.github.com/repos/kid1412z/smile/subscribers","subscription_url":"https://api.github.com/repos/kid1412z/smile/subscription","commits_url":"https://api.github.com/repos/kid1412z/smile/commits{/sha}","git_commits_url":"https://api.github.com/repos/kid1412z/smile/git/commits{/sha}","comments_url":"https://api.github.com/repos/kid1412z/smile/comments{/number}","issue_comment_url":"https://api.github.com/repos/kid1412z/smile/issues/comments{/number}","contents_url":"https://api.github.com/repos/kid1412z/smile/contents/{+path}","compare_url":"https://api.github.com/repos/kid1412z/smile/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kid1412z/smile/merges","archive_url":"https://api.github.com/repos/kid1412z/smile/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kid1412z/smile/downloads","issues_url":"https://api.github.com/repos/kid1412z/smile/issues{/number}","pulls_url":"https://api.github.com/repos/kid1412z/smile/pulls{/number}","milestones_url":"https://api.github.com/repos/kid1412z/smile/milestones{/number}","notifications_url":"https://api.github.com/repos/kid1412z/smile/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kid1412z/smile/labels{/name}","releases_url":"https://api.github.com/repos/kid1412z/smile/releases{/id}","created_at":"2015-03-23T14:48:20Z","updated_at":"2015-04-13T12:22:08Z","pushed_at":"2015-04-13T12:22:08Z","git_url":"git://github.com/kid1412z/smile.git","ssh_url":"git@github.com:kid1412z/smile.git","clone_url":"https://github.com/kid1412z/smile.git","svn_url":"https://github.com/kid1412z/smile","homepage":null,"size":27294,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"haifengl:master","ref":"master","sha":"157827b1b9dce3b10a93b96f1d33859edda6d699","user":{"login":"haifengl","id":5502052,"avatar_url":"https://avatars.githubusercontent.com/u/5502052?v=3","gravatar_id":"","url":"https://api.github.com/users/haifengl","html_url":"https://github.com/haifengl","followers_url":"https://api.github.com/users/haifengl/followers","following_url":"https://api.github.com/users/haifengl/following{/other_user}","gists_url":"https://api.github.com/users/haifengl/gists{/gist_id}","starred_url":"https://api.github.com/users/haifengl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haifengl/subscriptions","organizations_url":"https://api.github.com/users/haifengl/orgs","repos_url":"https://api.github.com/users/haifengl/repos","events_url":"https://api.github.com/users/haifengl/events{/privacy}","received_events_url":"https://api.github.com/users/haifengl/received_events","type":"User","site_admin":false},"repo":{"id":26921116,"name":"smile","full_name":"haifengl/smile","owner":{"login":"haifengl","id":5502052,"avatar_url":"https://avatars.githubusercontent.com/u/5502052?v=3","gravatar_id":"","url":"https://api.github.com/users/haifengl","html_url":"https://github.com/haifengl","followers_url":"https://api.github.com/users/haifengl/followers","following_url":"https://api.github.com/users/haifengl/following{/other_user}","gists_url":"https://api.github.com/users/haifengl/gists{/gist_id}","starred_url":"https://api.github.com/users/haifengl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haifengl/subscriptions","organizations_url":"https://api.github.com/users/haifengl/orgs","repos_url":"https://api.github.com/users/haifengl/repos","events_url":"https://api.github.com/users/haifengl/events{/privacy}","received_events_url":"https://api.github.com/users/haifengl/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/haifengl/smile","description":"Statistical Machine Intelligence & Learning Engine","fork":false,"url":"https://api.github.com/repos/haifengl/smile","forks_url":"https://api.github.com/repos/haifengl/smile/forks","keys_url":"https://api.github.com/repos/haifengl/smile/keys{/key_id}","collaborators_url":"https://api.github.com/repos/haifengl/smile/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/haifengl/smile/teams","hooks_url":"https://api.github.com/repos/haifengl/smile/hooks","issue_events_url":"https://api.github.com/repos/haifengl/smile/issues/events{/number}","events_url":"https://api.github.com/repos/haifengl/smile/events","assignees_url":"https://api.github.com/repos/haifengl/smile/assignees{/user}","branches_url":"https://api.github.com/repos/haifengl/smile/branches{/branch}","tags_url":"https://api.github.com/repos/haifengl/smile/tags","blobs_url":"https://api.github.com/repos/haifengl/smile/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/haifengl/smile/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/haifengl/smile/git/refs{/sha}","trees_url":"https://api.github.com/repos/haifengl/smile/git/trees{/sha}","statuses_url":"https://api.github.com/repos/haifengl/smile/statuses/{sha}","languages_url":"https://api.github.com/repos/haifengl/smile/languages","stargazers_url":"https://api.github.com/repos/haifengl/smile/stargazers","contributors_url":"https://api.github.com/repos/haifengl/smile/contributors","subscribers_url":"https://api.github.com/repos/haifengl/smile/subscribers","subscription_url":"https://api.github.com/repos/haifengl/smile/subscription","commits_url":"https://api.github.com/repos/haifengl/smile/commits{/sha}","git_commits_url":"https://api.github.com/repos/haifengl/smile/git/commits{/sha}","comments_url":"https://api.github.com/repos/haifengl/smile/comments{/number}","issue_comment_url":"https://api.github.com/repos/haifengl/smile/issues/comments{/number}","contents_url":"https://api.github.com/repos/haifengl/smile/contents/{+path}","compare_url":"https://api.github.com/repos/haifengl/smile/compare/{base}...{head}","merges_url":"https://api.github.com/repos/haifengl/smile/merges","archive_url":"https://api.github.com/repos/haifengl/smile/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/haifengl/smile/downloads","issues_url":"https://api.github.com/repos/haifengl/smile/issues{/number}","pulls_url":"https://api.github.com/repos/haifengl/smile/pulls{/number}","milestones_url":"https://api.github.com/repos/haifengl/smile/milestones{/number}","notifications_url":"https://api.github.com/repos/haifengl/smile/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/haifengl/smile/labels{/name}","releases_url":"https://api.github.com/repos/haifengl/smile/releases{/id}","created_at":"2014-11-20T16:28:12Z","updated_at":"2015-04-14T02:43:03Z","pushed_at":"2015-04-14T02:43:03Z","git_url":"git://github.com/haifengl/smile.git","ssh_url":"git@github.com:haifengl/smile.git","clone_url":"https://github.com/haifengl/smile.git","svn_url":"https://github.com/haifengl/smile","homepage":null,"size":27993,"stargazers_count":1483,"watchers_count":1483,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":110,"mirror_url":null,"open_issues_count":3,"forks":110,"open_issues":3,"watchers":1483,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/haifengl/smile/pulls/24"},"html":{"href":"https://github.com/haifengl/smile/pull/24"},"issue":{"href":"https://api.github.com/repos/haifengl/smile/issues/24"},"comments":{"href":"https://api.github.com/repos/haifengl/smile/issues/24/comments"},"review_comments":{"href":"https://api.github.com/repos/haifengl/smile/pulls/24/comments"},"review_comment":{"href":"https://api.github.com/repos/haifengl/smile/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/haifengl/smile/pulls/24/commits"},"statuses":{"href":"https://api.github.com/repos/haifengl/smile/statuses/f9ec487a7b8316e1de22ffc038ba459e26ba4676"}}}}