{"action":"created","comment":{"url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/comments/46609753","id":46609753,"diff_hunk":"@@ -187,11 +187,6 @@ def add_citation_counts():\n             es,\n             query={\n                 \"_source\": \"references.recid\",\n-                \"filter\": {\n-                    \"exists\": {\n-                        \"field\": \"references.recid\"\n-                    }\n-                },","path":"inspirehep/modules/migrator/tasks.py","position":8,"original_position":8,"commit_id":"6c6ae942f6d7af52e94c40bfcd3d537714d92bc9","original_commit_id":"6c6ae942f6d7af52e94c40bfcd3d537714d92bc9","user":{"login":"kaplun","id":119349,"avatar_url":"https://avatars.githubusercontent.com/u/119349?v=3","gravatar_id":"","url":"https://api.github.com/users/kaplun","html_url":"https://github.com/kaplun","followers_url":"https://api.github.com/users/kaplun/followers","following_url":"https://api.github.com/users/kaplun/following{/other_user}","gists_url":"https://api.github.com/users/kaplun/gists{/gist_id}","starred_url":"https://api.github.com/users/kaplun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaplun/subscriptions","organizations_url":"https://api.github.com/users/kaplun/orgs","repos_url":"https://api.github.com/users/kaplun/repos","events_url":"https://api.github.com/users/kaplun/events{/privacy}","received_events_url":"https://api.github.com/users/kaplun/received_events","type":"User","site_admin":false},"body":"I don't think this is necessary to solve the bug. Actually I realize the best way would really be to exploit the `citations_lookup`.\r\n\r\nYou should rather create the `records_to_update` list via a generator independent from the input.:\r\n```python\r\ndef get_records_to_update_generator(citation_lookup):\r\n    for recid, citation_count in citation_lookup.iteritems():\r\n        yield {'_op_type': 'update',\r\n                     '_index': 'hep',\r\n                     '_type': 'record',\r\n                     '_id': recid,\r\n                     'doc': {'citation_count': citation_count}\r\n                 }\r\n\r\nfor chunk in chunker(get_records_to_update_generator(citations_lookup), CHUNK_SIZE):\r\n    es_bulk(es, chunk)\r\n```\r\n\r\nWith the current implementation you were risking of filling up the memory with a `records_to_update` list with 1.5M records :smile: .","created_at":"2015-12-03T20:50:02Z","updated_at":"2015-12-03T20:50:02Z","html_url":"https://github.com/inspirehep/inspire-next/pull/610#discussion_r46609753","pull_request_url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610","_links":{"self":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/comments/46609753"},"html":{"href":"https://github.com/inspirehep/inspire-next/pull/610#discussion_r46609753"},"pull_request":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610"}}},"pull_request":{"url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610","id":52551145,"html_url":"https://github.com/inspirehep/inspire-next/pull/610","diff_url":"https://github.com/inspirehep/inspire-next/pull/610.diff","patch_url":"https://github.com/inspirehep/inspire-next/pull/610.patch","issue_url":"https://api.github.com/repos/inspirehep/inspire-next/issues/610","number":610,"state":"open","locked":false,"title":"citations: citation_count fix","user":{"login":"tomaszgy","id":10139760,"avatar_url":"https://avatars.githubusercontent.com/u/10139760?v=3","gravatar_id":"","url":"https://api.github.com/users/tomaszgy","html_url":"https://github.com/tomaszgy","followers_url":"https://api.github.com/users/tomaszgy/followers","following_url":"https://api.github.com/users/tomaszgy/following{/other_user}","gists_url":"https://api.github.com/users/tomaszgy/gists{/gist_id}","starred_url":"https://api.github.com/users/tomaszgy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomaszgy/subscriptions","organizations_url":"https://api.github.com/users/tomaszgy/orgs","repos_url":"https://api.github.com/users/tomaszgy/repos","events_url":"https://api.github.com/users/tomaszgy/events{/privacy}","received_events_url":"https://api.github.com/users/tomaszgy/received_events","type":"User","site_admin":false},"body":"* add citation_count to all records, not only those that have references\r\n\r\nSigned-off-by: Tomasz <tomaszgy@gmail.com>","created_at":"2015-12-03T15:45:46Z","updated_at":"2015-12-03T20:50:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b08315a4bc7dcf341769d440e3a63bb589f4bb92","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610/commits","review_comments_url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610/comments","review_comment_url":"https://api.github.com/repos/inspirehep/inspire-next/pulls/comments{/number}","comments_url":"https://api.github.com/repos/inspirehep/inspire-next/issues/610/comments","statuses_url":"https://api.github.com/repos/inspirehep/inspire-next/statuses/6c6ae942f6d7af52e94c40bfcd3d537714d92bc9","head":{"label":"tomaszgy:citations_migrator","ref":"citations_migrator","sha":"6c6ae942f6d7af52e94c40bfcd3d537714d92bc9","user":{"login":"tomaszgy","id":10139760,"avatar_url":"https://avatars.githubusercontent.com/u/10139760?v=3","gravatar_id":"","url":"https://api.github.com/users/tomaszgy","html_url":"https://github.com/tomaszgy","followers_url":"https://api.github.com/users/tomaszgy/followers","following_url":"https://api.github.com/users/tomaszgy/following{/other_user}","gists_url":"https://api.github.com/users/tomaszgy/gists{/gist_id}","starred_url":"https://api.github.com/users/tomaszgy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomaszgy/subscriptions","organizations_url":"https://api.github.com/users/tomaszgy/orgs","repos_url":"https://api.github.com/users/tomaszgy/repos","events_url":"https://api.github.com/users/tomaszgy/events{/privacy}","received_events_url":"https://api.github.com/users/tomaszgy/received_events","type":"User","site_admin":false},"repo":{"id":45853625,"name":"inspire-next","full_name":"tomaszgy/inspire-next","owner":{"login":"tomaszgy","id":10139760,"avatar_url":"https://avatars.githubusercontent.com/u/10139760?v=3","gravatar_id":"","url":"https://api.github.com/users/tomaszgy","html_url":"https://github.com/tomaszgy","followers_url":"https://api.github.com/users/tomaszgy/followers","following_url":"https://api.github.com/users/tomaszgy/following{/other_user}","gists_url":"https://api.github.com/users/tomaszgy/gists{/gist_id}","starred_url":"https://api.github.com/users/tomaszgy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomaszgy/subscriptions","organizations_url":"https://api.github.com/users/tomaszgy/orgs","repos_url":"https://api.github.com/users/tomaszgy/repos","events_url":"https://api.github.com/users/tomaszgy/events{/privacy}","received_events_url":"https://api.github.com/users/tomaszgy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tomaszgy/inspire-next","description":"INSPIRE overlay repository to be used with https://github.com/inspirehep/invenio@labs branch","fork":true,"url":"https://api.github.com/repos/tomaszgy/inspire-next","forks_url":"https://api.github.com/repos/tomaszgy/inspire-next/forks","keys_url":"https://api.github.com/repos/tomaszgy/inspire-next/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tomaszgy/inspire-next/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tomaszgy/inspire-next/teams","hooks_url":"https://api.github.com/repos/tomaszgy/inspire-next/hooks","issue_events_url":"https://api.github.com/repos/tomaszgy/inspire-next/issues/events{/number}","events_url":"https://api.github.com/repos/tomaszgy/inspire-next/events","assignees_url":"https://api.github.com/repos/tomaszgy/inspire-next/assignees{/user}","branches_url":"https://api.github.com/repos/tomaszgy/inspire-next/branches{/branch}","tags_url":"https://api.github.com/repos/tomaszgy/inspire-next/tags","blobs_url":"https://api.github.com/repos/tomaszgy/inspire-next/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tomaszgy/inspire-next/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tomaszgy/inspire-next/git/refs{/sha}","trees_url":"https://api.github.com/repos/tomaszgy/inspire-next/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tomaszgy/inspire-next/statuses/{sha}","languages_url":"https://api.github.com/repos/tomaszgy/inspire-next/languages","stargazers_url":"https://api.github.com/repos/tomaszgy/inspire-next/stargazers","contributors_url":"https://api.github.com/repos/tomaszgy/inspire-next/contributors","subscribers_url":"https://api.github.com/repos/tomaszgy/inspire-next/subscribers","subscription_url":"https://api.github.com/repos/tomaszgy/inspire-next/subscription","commits_url":"https://api.github.com/repos/tomaszgy/inspire-next/commits{/sha}","git_commits_url":"https://api.github.com/repos/tomaszgy/inspire-next/git/commits{/sha}","comments_url":"https://api.github.com/repos/tomaszgy/inspire-next/comments{/number}","issue_comment_url":"https://api.github.com/repos/tomaszgy/inspire-next/issues/comments{/number}","contents_url":"https://api.github.com/repos/tomaszgy/inspire-next/contents/{+path}","compare_url":"https://api.github.com/repos/tomaszgy/inspire-next/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tomaszgy/inspire-next/merges","archive_url":"https://api.github.com/repos/tomaszgy/inspire-next/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tomaszgy/inspire-next/downloads","issues_url":"https://api.github.com/repos/tomaszgy/inspire-next/issues{/number}","pulls_url":"https://api.github.com/repos/tomaszgy/inspire-next/pulls{/number}","milestones_url":"https://api.github.com/repos/tomaszgy/inspire-next/milestones{/number}","notifications_url":"https://api.github.com/repos/tomaszgy/inspire-next/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tomaszgy/inspire-next/labels{/name}","releases_url":"https://api.github.com/repos/tomaszgy/inspire-next/releases{/id}","created_at":"2015-11-09T17:13:36Z","updated_at":"2015-11-09T17:13:43Z","pushed_at":"2015-12-03T15:45:00Z","git_url":"git://github.com/tomaszgy/inspire-next.git","ssh_url":"git@github.com:tomaszgy/inspire-next.git","clone_url":"https://github.com/tomaszgy/inspire-next.git","svn_url":"https://github.com/tomaszgy/inspire-next","homepage":"https://labs.inspirehep.net","size":33417,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"inspirehep:master","ref":"master","sha":"d6128ad403942a5c16cb72ed20e3044357ae29f6","user":{"login":"inspirehep","id":356664,"avatar_url":"https://avatars.githubusercontent.com/u/356664?v=3","gravatar_id":"","url":"https://api.github.com/users/inspirehep","html_url":"https://github.com/inspirehep","followers_url":"https://api.github.com/users/inspirehep/followers","following_url":"https://api.github.com/users/inspirehep/following{/other_user}","gists_url":"https://api.github.com/users/inspirehep/gists{/gist_id}","starred_url":"https://api.github.com/users/inspirehep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inspirehep/subscriptions","organizations_url":"https://api.github.com/users/inspirehep/orgs","repos_url":"https://api.github.com/users/inspirehep/repos","events_url":"https://api.github.com/users/inspirehep/events{/privacy}","received_events_url":"https://api.github.com/users/inspirehep/received_events","type":"Organization","site_admin":false},"repo":{"id":15221515,"name":"inspire-next","full_name":"inspirehep/inspire-next","owner":{"login":"inspirehep","id":356664,"avatar_url":"https://avatars.githubusercontent.com/u/356664?v=3","gravatar_id":"","url":"https://api.github.com/users/inspirehep","html_url":"https://github.com/inspirehep","followers_url":"https://api.github.com/users/inspirehep/followers","following_url":"https://api.github.com/users/inspirehep/following{/other_user}","gists_url":"https://api.github.com/users/inspirehep/gists{/gist_id}","starred_url":"https://api.github.com/users/inspirehep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inspirehep/subscriptions","organizations_url":"https://api.github.com/users/inspirehep/orgs","repos_url":"https://api.github.com/users/inspirehep/repos","events_url":"https://api.github.com/users/inspirehep/events{/privacy}","received_events_url":"https://api.github.com/users/inspirehep/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/inspirehep/inspire-next","description":"INSPIRE overlay repository to be used with https://github.com/inspirehep/invenio@labs branch","fork":false,"url":"https://api.github.com/repos/inspirehep/inspire-next","forks_url":"https://api.github.com/repos/inspirehep/inspire-next/forks","keys_url":"https://api.github.com/repos/inspirehep/inspire-next/keys{/key_id}","collaborators_url":"https://api.github.com/repos/inspirehep/inspire-next/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/inspirehep/inspire-next/teams","hooks_url":"https://api.github.com/repos/inspirehep/inspire-next/hooks","issue_events_url":"https://api.github.com/repos/inspirehep/inspire-next/issues/events{/number}","events_url":"https://api.github.com/repos/inspirehep/inspire-next/events","assignees_url":"https://api.github.com/repos/inspirehep/inspire-next/assignees{/user}","branches_url":"https://api.github.com/repos/inspirehep/inspire-next/branches{/branch}","tags_url":"https://api.github.com/repos/inspirehep/inspire-next/tags","blobs_url":"https://api.github.com/repos/inspirehep/inspire-next/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/inspirehep/inspire-next/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/inspirehep/inspire-next/git/refs{/sha}","trees_url":"https://api.github.com/repos/inspirehep/inspire-next/git/trees{/sha}","statuses_url":"https://api.github.com/repos/inspirehep/inspire-next/statuses/{sha}","languages_url":"https://api.github.com/repos/inspirehep/inspire-next/languages","stargazers_url":"https://api.github.com/repos/inspirehep/inspire-next/stargazers","contributors_url":"https://api.github.com/repos/inspirehep/inspire-next/contributors","subscribers_url":"https://api.github.com/repos/inspirehep/inspire-next/subscribers","subscription_url":"https://api.github.com/repos/inspirehep/inspire-next/subscription","commits_url":"https://api.github.com/repos/inspirehep/inspire-next/commits{/sha}","git_commits_url":"https://api.github.com/repos/inspirehep/inspire-next/git/commits{/sha}","comments_url":"https://api.github.com/repos/inspirehep/inspire-next/comments{/number}","issue_comment_url":"https://api.github.com/repos/inspirehep/inspire-next/issues/comments{/number}","contents_url":"https://api.github.com/repos/inspirehep/inspire-next/contents/{+path}","compare_url":"https://api.github.com/repos/inspirehep/inspire-next/compare/{base}...{head}","merges_url":"https://api.github.com/repos/inspirehep/inspire-next/merges","archive_url":"https://api.github.com/repos/inspirehep/inspire-next/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/inspirehep/inspire-next/downloads","issues_url":"https://api.github.com/repos/inspirehep/inspire-next/issues{/number}","pulls_url":"https://api.github.com/repos/inspirehep/inspire-next/pulls{/number}","milestones_url":"https://api.github.com/repos/inspirehep/inspire-next/milestones{/number}","notifications_url":"https://api.github.com/repos/inspirehep/inspire-next/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/inspirehep/inspire-next/labels{/name}","releases_url":"https://api.github.com/repos/inspirehep/inspire-next/releases{/id}","created_at":"2013-12-16T09:11:58Z","updated_at":"2015-10-13T09:48:55Z","pushed_at":"2015-12-03T16:28:12Z","git_url":"git://github.com/inspirehep/inspire-next.git","ssh_url":"git@github.com:inspirehep/inspire-next.git","clone_url":"https://github.com/inspirehep/inspire-next.git","svn_url":"https://github.com/inspirehep/inspire-next","homepage":"https://labs.inspirehep.net","size":37837,"stargazers_count":8,"watchers_count":8,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":23,"mirror_url":null,"open_issues_count":94,"forks":23,"open_issues":94,"watchers":8,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610"},"html":{"href":"https://github.com/inspirehep/inspire-next/pull/610"},"issue":{"href":"https://api.github.com/repos/inspirehep/inspire-next/issues/610"},"comments":{"href":"https://api.github.com/repos/inspirehep/inspire-next/issues/610/comments"},"review_comments":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610/comments"},"review_comment":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/inspirehep/inspire-next/pulls/610/commits"},"statuses":{"href":"https://api.github.com/repos/inspirehep/inspire-next/statuses/6c6ae942f6d7af52e94c40bfcd3d537714d92bc9"}}}}