{"action":"closed","number":385,"pull_request":{"url":"https://api.github.com/repos/piskvorky/gensim/pulls/385","id":39228774,"html_url":"https://github.com/piskvorky/gensim/pull/385","diff_url":"https://github.com/piskvorky/gensim/pull/385.diff","patch_url":"https://github.com/piskvorky/gensim/pull/385.patch","issue_url":"https://api.github.com/repos/piskvorky/gensim/issues/385","number":385,"state":"closed","locked":false,"title":"Prune vocab","user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"body":"Use less memory during word2vec and doc2vec vocabulary building.\r\n\r\nRefactored Phrases to use the same function for pruning the count dictionary, which was moved under `utils.prune_vocab`.\r\n\r\nThe main win comes from keeping raw counts in a plain dict (defaultdict) and only creating the expensive Vocab objects for words with count >= `min_count`.\r\n\r\nThe additional option of trimming the defaultdict to be <= `max_vocab_size` is likely not even necessary (I set the default `max_vocab_size=10,000,000`, which consumes only about 1GB of RAM).\r\n\r\nI also removed the stderr flushes (belongs to logging, not core gensim -- logs can go to files etc, stderr is just incidental) and made some minor PEP8 fixes.","created_at":"2015-07-05T23:28:23Z","updated_at":"2015-07-06T14:58:00Z","closed_at":"2015-07-06T14:58:00Z","merged_at":"2015-07-06T14:58:00Z","merge_commit_sha":"333775830d264b99d1bf4b5dac9a541df2c6c9b7","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/piskvorky/gensim/pulls/385/commits","review_comments_url":"https://api.github.com/repos/piskvorky/gensim/pulls/385/comments","review_comment_url":"https://api.github.com/repos/piskvorky/gensim/pulls/comments{/number}","comments_url":"https://api.github.com/repos/piskvorky/gensim/issues/385/comments","statuses_url":"https://api.github.com/repos/piskvorky/gensim/statuses/6a8faa2f27053e43516f9cbe2f4d0bfcee017db3","head":{"label":"piskvorky:prune_vocab","ref":"prune_vocab","sha":"6a8faa2f27053e43516f9cbe2f4d0bfcee017db3","user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"repo":{"id":1349775,"name":"gensim","full_name":"piskvorky/gensim","owner":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/piskvorky/gensim","description":"Topic Modelling for Humans","fork":false,"url":"https://api.github.com/repos/piskvorky/gensim","forks_url":"https://api.github.com/repos/piskvorky/gensim/forks","keys_url":"https://api.github.com/repos/piskvorky/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/piskvorky/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/piskvorky/gensim/teams","hooks_url":"https://api.github.com/repos/piskvorky/gensim/hooks","issue_events_url":"https://api.github.com/repos/piskvorky/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/piskvorky/gensim/events","assignees_url":"https://api.github.com/repos/piskvorky/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/piskvorky/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/piskvorky/gensim/tags","blobs_url":"https://api.github.com/repos/piskvorky/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/piskvorky/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/piskvorky/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/piskvorky/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/piskvorky/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/piskvorky/gensim/languages","stargazers_url":"https://api.github.com/repos/piskvorky/gensim/stargazers","contributors_url":"https://api.github.com/repos/piskvorky/gensim/contributors","subscribers_url":"https://api.github.com/repos/piskvorky/gensim/subscribers","subscription_url":"https://api.github.com/repos/piskvorky/gensim/subscription","commits_url":"https://api.github.com/repos/piskvorky/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/piskvorky/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/piskvorky/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/piskvorky/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/piskvorky/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/piskvorky/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/piskvorky/gensim/merges","archive_url":"https://api.github.com/repos/piskvorky/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/piskvorky/gensim/downloads","issues_url":"https://api.github.com/repos/piskvorky/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/piskvorky/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/piskvorky/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/piskvorky/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/piskvorky/gensim/labels{/name}","releases_url":"https://api.github.com/repos/piskvorky/gensim/releases{/id}","created_at":"2011-02-10T07:43:04Z","updated_at":"2015-07-06T09:10:49Z","pushed_at":"2015-07-06T14:58:00Z","git_url":"git://github.com/piskvorky/gensim.git","ssh_url":"git@github.com:piskvorky/gensim.git","clone_url":"https://github.com/piskvorky/gensim.git","svn_url":"https://github.com/piskvorky/gensim","homepage":"http://radimrehurek.com/gensim/","size":20127,"stargazers_count":1547,"watchers_count":1547,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":541,"mirror_url":null,"open_issues_count":72,"forks":541,"open_issues":72,"watchers":1547,"default_branch":"develop"}},"base":{"label":"piskvorky:develop","ref":"develop","sha":"77d4def7728742674e33c5014c3afee89f4b40fb","user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"repo":{"id":1349775,"name":"gensim","full_name":"piskvorky/gensim","owner":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/piskvorky/gensim","description":"Topic Modelling for Humans","fork":false,"url":"https://api.github.com/repos/piskvorky/gensim","forks_url":"https://api.github.com/repos/piskvorky/gensim/forks","keys_url":"https://api.github.com/repos/piskvorky/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/piskvorky/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/piskvorky/gensim/teams","hooks_url":"https://api.github.com/repos/piskvorky/gensim/hooks","issue_events_url":"https://api.github.com/repos/piskvorky/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/piskvorky/gensim/events","assignees_url":"https://api.github.com/repos/piskvorky/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/piskvorky/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/piskvorky/gensim/tags","blobs_url":"https://api.github.com/repos/piskvorky/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/piskvorky/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/piskvorky/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/piskvorky/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/piskvorky/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/piskvorky/gensim/languages","stargazers_url":"https://api.github.com/repos/piskvorky/gensim/stargazers","contributors_url":"https://api.github.com/repos/piskvorky/gensim/contributors","subscribers_url":"https://api.github.com/repos/piskvorky/gensim/subscribers","subscription_url":"https://api.github.com/repos/piskvorky/gensim/subscription","commits_url":"https://api.github.com/repos/piskvorky/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/piskvorky/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/piskvorky/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/piskvorky/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/piskvorky/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/piskvorky/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/piskvorky/gensim/merges","archive_url":"https://api.github.com/repos/piskvorky/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/piskvorky/gensim/downloads","issues_url":"https://api.github.com/repos/piskvorky/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/piskvorky/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/piskvorky/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/piskvorky/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/piskvorky/gensim/labels{/name}","releases_url":"https://api.github.com/repos/piskvorky/gensim/releases{/id}","created_at":"2011-02-10T07:43:04Z","updated_at":"2015-07-06T09:10:49Z","pushed_at":"2015-07-06T14:58:00Z","git_url":"git://github.com/piskvorky/gensim.git","ssh_url":"git@github.com:piskvorky/gensim.git","clone_url":"https://github.com/piskvorky/gensim.git","svn_url":"https://github.com/piskvorky/gensim","homepage":"http://radimrehurek.com/gensim/","size":20127,"stargazers_count":1547,"watchers_count":1547,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":541,"mirror_url":null,"open_issues_count":72,"forks":541,"open_issues":72,"watchers":1547,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/385"},"html":{"href":"https://github.com/piskvorky/gensim/pull/385"},"issue":{"href":"https://api.github.com/repos/piskvorky/gensim/issues/385"},"comments":{"href":"https://api.github.com/repos/piskvorky/gensim/issues/385/comments"},"review_comments":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/385/comments"},"review_comment":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/piskvorky/gensim/pulls/385/commits"},"statuses":{"href":"https://api.github.com/repos/piskvorky/gensim/statuses/6a8faa2f27053e43516f9cbe2f4d0bfcee017db3"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"comments":2,"review_comments":11,"commits":5,"additions":138,"deletions":123,"changed_files":6}}