{"action":"created","comment":{"url":"https://api.github.com/repos/piskvorky/gensim/pulls/comments/43835633","id":43835633,"diff_hunk":"@@ -106,17 +206,23 @@ def __init__(self, sentences=None, min_count=5, threshold=10.0,\n         `delimiter` is the glue character used to join collocation tokens, and\n         should be a byte string (e.g. b'_').\n \n+        `average_sentence_size`: expected number of tokens in average sentence.\n+            max_vocab_size / average_sentence_size sentences is used to build python\n+            dictionary per thread.\n+            Lowering `average_sentence_size` will increase memory demands.\n         \"\"\"\n         if min_count <= 0:\n-            raise ValueError(\"min_count should be at least 1\")\n-\n+            raise ValueError(\"min_count should be 1\")\n+        if min_count > 1:\n+            logger.warning(\"min_count should be 1\")","path":"gensim/models/phrases.py","position":153,"original_position":153,"commit_id":"a21e3b5c4628a555a716b8d28228edb4256356c1","original_commit_id":"a21e3b5c4628a555a716b8d28228edb4256356c1","user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"body":"Agreed with @gojomo -- we don't want to keep dead code around.\r\n\r\nAnd in case of `Phrases`, we don't need the \"old\" behaviour either. The Phrases functionality is new and not widely used yet, so a clear release note saying things have changed seems enough here.","created_at":"2015-11-04T02:15:02Z","updated_at":"2015-11-04T02:15:02Z","html_url":"https://github.com/piskvorky/gensim/pull/508#discussion_r43835633","pull_request_url":"https://api.github.com/repos/piskvorky/gensim/pulls/508","_links":{"self":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/comments/43835633"},"html":{"href":"https://github.com/piskvorky/gensim/pull/508#discussion_r43835633"},"pull_request":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/508"}}},"pull_request":{"url":"https://api.github.com/repos/piskvorky/gensim/pulls/508","id":49526396,"html_url":"https://github.com/piskvorky/gensim/pull/508","diff_url":"https://github.com/piskvorky/gensim/pull/508.diff","patch_url":"https://github.com/piskvorky/gensim/pull/508.patch","issue_url":"https://api.github.com/repos/piskvorky/gensim/issues/508","number":508,"state":"open","locked":false,"title":"Phrases: replace default dict by combination of hyperloglog and CountMinSketch alg.","user":{"login":"janrygl","id":11831178,"avatar_url":"https://avatars.githubusercontent.com/u/11831178?v=3","gravatar_id":"","url":"https://api.github.com/users/janrygl","html_url":"https://github.com/janrygl","followers_url":"https://api.github.com/users/janrygl/followers","following_url":"https://api.github.com/users/janrygl/following{/other_user}","gists_url":"https://api.github.com/users/janrygl/gists{/gist_id}","starred_url":"https://api.github.com/users/janrygl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janrygl/subscriptions","organizations_url":"https://api.github.com/users/janrygl/orgs","repos_url":"https://api.github.com/users/janrygl/repos","events_url":"https://api.github.com/users/janrygl/events{/privacy}","received_events_url":"https://api.github.com/users/janrygl/received_events","type":"User","site_admin":false},"body":"### Changes:\r\n * Pruning words with lower frequencies is removed.\r\n * Approximative algorithms are used:\r\n   * `hyperloglog` for counting vocabulary size (default vocabulary size error 1%)\r\n   * `CountMinSketch` for frequency count (if F is true freq, result is in range [F, F * (1+e)], default e = 0.01)\r\n * Uses `defaultdicts` for chunk processing (it can be rewritten to parallel chunk building)\r\n * May need to update constants:\r\n   * threshold can be lower with higher vocabulary size\r\n   * chunks can be bigger (sentence is expected to contain 20 words, can be 10)\r\n   * allowed errors can be lower\r\n\r\n### Comparison of old and new implementation:\r\n\r\n * 100 000 000 randomly generated sentences (10 % stop words, 90 % random)\r\n * max_vocab_size=40000000 (ignored in new implementation)\r\n\r\nImplementation | Old | New\r\n---- | ---- | ----\r\nTime (h) | 6 h 9 min | 28 h 26 min |\r\nTime (ratio) | 100% | 462 % |\r\nVocabulary size | 29 700 928 | 761 363 290\r\n\r\n\r\n * 20 000 000 randomly generated sentences (10 % stop words, 90 % random)\r\n * max_vocab_size=40000000 (ignored in new implementation)\r\n\r\nImplementation | Old | New\r\n---- | ---- | ----\r\nTime (h) | 2h 26 min | 11 h 26 min |\r\nTime (ratio) | 100% | 469 % |\r\nVocabulary size | 12 692 683 | 309 776 447\r\n\r\n\r\n * 10 000 000 randomly generated sentences (10 % stop words, 90 % random)\r\n * max_vocab_size=40000000 (ignored in new implementation)\r\n\r\nImplementation | Old | New\r\n---- | ---- | ----\r\nTime (h) | 36 min | 2 h 53 min \r\nTime (ratio) | 100% | 480 % \r\nVocabulary size | 4 077 048 | 81 486 060\r\n\r\n\r\n * from nltk.corpus import gutenberg\r\n * guttenber.sents() (98551 sentences)\r\n\r\nImplementation | Old | New\r\n---- | ---- | ----\r\nTime (s) | 14 s | 34 s \r\nTime (ratio) | 100% | 240 % \r\nVocabulary size | 618 421 | 619 564\r\n\r\n * from nltk.corpus import brown\r\n * brown.sents()  57340 sentences)\r\n\r\nImplementation | Old | New\r\n---- | ---- | ----\r\nTime (s) | 7 s | 24 s \r\nTime (ratio) | 100% | 340 % \r\nVocabulary size | 503 281 | 504 632","created_at":"2015-11-03T07:37:46Z","updated_at":"2015-11-04T02:15:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a85360c7ca7dbe449ca9892c0de2cfabc5defc74","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/piskvorky/gensim/pulls/508/commits","review_comments_url":"https://api.github.com/repos/piskvorky/gensim/pulls/508/comments","review_comment_url":"https://api.github.com/repos/piskvorky/gensim/pulls/comments{/number}","comments_url":"https://api.github.com/repos/piskvorky/gensim/issues/508/comments","statuses_url":"https://api.github.com/repos/piskvorky/gensim/statuses/a21e3b5c4628a555a716b8d28228edb4256356c1","head":{"label":"janrygl:develop","ref":"develop","sha":"a21e3b5c4628a555a716b8d28228edb4256356c1","user":{"login":"janrygl","id":11831178,"avatar_url":"https://avatars.githubusercontent.com/u/11831178?v=3","gravatar_id":"","url":"https://api.github.com/users/janrygl","html_url":"https://github.com/janrygl","followers_url":"https://api.github.com/users/janrygl/followers","following_url":"https://api.github.com/users/janrygl/following{/other_user}","gists_url":"https://api.github.com/users/janrygl/gists{/gist_id}","starred_url":"https://api.github.com/users/janrygl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janrygl/subscriptions","organizations_url":"https://api.github.com/users/janrygl/orgs","repos_url":"https://api.github.com/users/janrygl/repos","events_url":"https://api.github.com/users/janrygl/events{/privacy}","received_events_url":"https://api.github.com/users/janrygl/received_events","type":"User","site_admin":false},"repo":{"id":45182791,"name":"gensim","full_name":"janrygl/gensim","owner":{"login":"janrygl","id":11831178,"avatar_url":"https://avatars.githubusercontent.com/u/11831178?v=3","gravatar_id":"","url":"https://api.github.com/users/janrygl","html_url":"https://github.com/janrygl","followers_url":"https://api.github.com/users/janrygl/followers","following_url":"https://api.github.com/users/janrygl/following{/other_user}","gists_url":"https://api.github.com/users/janrygl/gists{/gist_id}","starred_url":"https://api.github.com/users/janrygl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janrygl/subscriptions","organizations_url":"https://api.github.com/users/janrygl/orgs","repos_url":"https://api.github.com/users/janrygl/repos","events_url":"https://api.github.com/users/janrygl/events{/privacy}","received_events_url":"https://api.github.com/users/janrygl/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/janrygl/gensim","description":"Topic Modelling for Humans","fork":true,"url":"https://api.github.com/repos/janrygl/gensim","forks_url":"https://api.github.com/repos/janrygl/gensim/forks","keys_url":"https://api.github.com/repos/janrygl/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/janrygl/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/janrygl/gensim/teams","hooks_url":"https://api.github.com/repos/janrygl/gensim/hooks","issue_events_url":"https://api.github.com/repos/janrygl/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/janrygl/gensim/events","assignees_url":"https://api.github.com/repos/janrygl/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/janrygl/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/janrygl/gensim/tags","blobs_url":"https://api.github.com/repos/janrygl/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/janrygl/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/janrygl/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/janrygl/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/janrygl/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/janrygl/gensim/languages","stargazers_url":"https://api.github.com/repos/janrygl/gensim/stargazers","contributors_url":"https://api.github.com/repos/janrygl/gensim/contributors","subscribers_url":"https://api.github.com/repos/janrygl/gensim/subscribers","subscription_url":"https://api.github.com/repos/janrygl/gensim/subscription","commits_url":"https://api.github.com/repos/janrygl/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/janrygl/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/janrygl/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/janrygl/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/janrygl/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/janrygl/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/janrygl/gensim/merges","archive_url":"https://api.github.com/repos/janrygl/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/janrygl/gensim/downloads","issues_url":"https://api.github.com/repos/janrygl/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/janrygl/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/janrygl/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/janrygl/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/janrygl/gensim/labels{/name}","releases_url":"https://api.github.com/repos/janrygl/gensim/releases{/id}","created_at":"2015-10-29T12:51:14Z","updated_at":"2015-10-29T12:51:17Z","pushed_at":"2015-11-03T09:36:49Z","git_url":"git://github.com/janrygl/gensim.git","ssh_url":"git@github.com:janrygl/gensim.git","clone_url":"https://github.com/janrygl/gensim.git","svn_url":"https://github.com/janrygl/gensim","homepage":"http://radimrehurek.com/gensim/","size":21491,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"piskvorky:develop","ref":"develop","sha":"e30666c65c391be4aa9d47514350530cd28d9e5c","user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"repo":{"id":1349775,"name":"gensim","full_name":"piskvorky/gensim","owner":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/piskvorky/gensim","description":"Topic Modelling for Humans","fork":false,"url":"https://api.github.com/repos/piskvorky/gensim","forks_url":"https://api.github.com/repos/piskvorky/gensim/forks","keys_url":"https://api.github.com/repos/piskvorky/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/piskvorky/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/piskvorky/gensim/teams","hooks_url":"https://api.github.com/repos/piskvorky/gensim/hooks","issue_events_url":"https://api.github.com/repos/piskvorky/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/piskvorky/gensim/events","assignees_url":"https://api.github.com/repos/piskvorky/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/piskvorky/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/piskvorky/gensim/tags","blobs_url":"https://api.github.com/repos/piskvorky/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/piskvorky/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/piskvorky/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/piskvorky/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/piskvorky/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/piskvorky/gensim/languages","stargazers_url":"https://api.github.com/repos/piskvorky/gensim/stargazers","contributors_url":"https://api.github.com/repos/piskvorky/gensim/contributors","subscribers_url":"https://api.github.com/repos/piskvorky/gensim/subscribers","subscription_url":"https://api.github.com/repos/piskvorky/gensim/subscription","commits_url":"https://api.github.com/repos/piskvorky/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/piskvorky/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/piskvorky/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/piskvorky/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/piskvorky/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/piskvorky/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/piskvorky/gensim/merges","archive_url":"https://api.github.com/repos/piskvorky/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/piskvorky/gensim/downloads","issues_url":"https://api.github.com/repos/piskvorky/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/piskvorky/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/piskvorky/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/piskvorky/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/piskvorky/gensim/labels{/name}","releases_url":"https://api.github.com/repos/piskvorky/gensim/releases{/id}","created_at":"2011-02-10T07:43:04Z","updated_at":"2015-11-03T21:58:11Z","pushed_at":"2015-11-03T18:55:52Z","git_url":"git://github.com/piskvorky/gensim.git","ssh_url":"git@github.com:piskvorky/gensim.git","clone_url":"https://github.com/piskvorky/gensim.git","svn_url":"https://github.com/piskvorky/gensim","homepage":"http://radimrehurek.com/gensim/","size":21491,"stargazers_count":1886,"watchers_count":1886,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":676,"mirror_url":null,"open_issues_count":115,"forks":676,"open_issues":115,"watchers":1886,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/508"},"html":{"href":"https://github.com/piskvorky/gensim/pull/508"},"issue":{"href":"https://api.github.com/repos/piskvorky/gensim/issues/508"},"comments":{"href":"https://api.github.com/repos/piskvorky/gensim/issues/508/comments"},"review_comments":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/508/comments"},"review_comment":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/508/commits"},"statuses":{"href":"https://api.github.com/repos/piskvorky/gensim/statuses/a21e3b5c4628a555a716b8d28228edb4256356c1"}}}}