{"action":"created","comment":{"url":"https://api.github.com/repos/alphagov/e-petitions/pulls/comments/35738963","id":35738963,"diff_hunk":"@@ -1,4 +1,8 @@\n RSpec.configure do |config|\n   config.include(EmailSpec::Helpers)\n   config.include(EmailSpec::Matchers)\n+\n+  config.before(:each) do\n+    ActionMailer::Base.deliveries.clear\n+  end","path":"spec/support/email.rb","position":7,"original_position":7,"commit_id":"2b317b570ee8a1984cf07c57b77931d7db81b65b","original_commit_id":"2b317b570ee8a1984cf07c57b77931d7db81b65b","user":{"login":"ansonK","id":722405,"avatar_url":"https://avatars.githubusercontent.com/u/722405?v=3","gravatar_id":"","url":"https://api.github.com/users/ansonK","html_url":"https://github.com/ansonK","followers_url":"https://api.github.com/users/ansonK/followers","following_url":"https://api.github.com/users/ansonK/following{/other_user}","gists_url":"https://api.github.com/users/ansonK/gists{/gist_id}","starred_url":"https://api.github.com/users/ansonK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ansonK/subscriptions","organizations_url":"https://api.github.com/users/ansonK/orgs","repos_url":"https://api.github.com/users/ansonK/repos","events_url":"https://api.github.com/users/ansonK/events{/privacy}","received_events_url":"https://api.github.com/users/ansonK/received_events","type":"User","site_admin":false},"body":"Started doing this but quickly realised its a far bigger job - most controller specs make assertions about emails sent and none of them use the email_spec helpers.\r\n\r\nRefactoring all of them would be a PR in itself ","created_at":"2015-07-29T08:48:31Z","updated_at":"2015-07-29T08:48:31Z","html_url":"https://github.com/alphagov/e-petitions/pull/360#discussion_r35738963","pull_request_url":"https://api.github.com/repos/alphagov/e-petitions/pulls/360","_links":{"self":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/comments/35738963"},"html":{"href":"https://github.com/alphagov/e-petitions/pull/360#discussion_r35738963"},"pull_request":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/360"}}},"pull_request":{"url":"https://api.github.com/repos/alphagov/e-petitions/pulls/360","id":41018340,"html_url":"https://github.com/alphagov/e-petitions/pull/360","diff_url":"https://github.com/alphagov/e-petitions/pull/360.diff","patch_url":"https://github.com/alphagov/e-petitions/pull/360.patch","issue_url":"https://api.github.com/repos/alphagov/e-petitions/issues/360","number":360,"state":"open","locked":false,"title":"Refactor the jobs that send email.","user":{"login":"ansonK","id":722405,"avatar_url":"https://avatars.githubusercontent.com/u/722405?v=3","gravatar_id":"","url":"https://api.github.com/users/ansonK","html_url":"https://github.com/ansonK","followers_url":"https://api.github.com/users/ansonK/followers","following_url":"https://api.github.com/users/ansonK/following{/other_user}","gists_url":"https://api.github.com/users/ansonK/gists{/gist_id}","starred_url":"https://api.github.com/users/ansonK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ansonK/subscriptions","organizations_url":"https://api.github.com/users/ansonK/orgs","repos_url":"https://api.github.com/users/ansonK/repos","events_url":"https://api.github.com/users/ansonK/events{/privacy}","received_events_url":"https://api.github.com/users/ansonK/received_events","type":"User","site_admin":false},"body":"Break the jobs that deliver email updates to petition signatories into 2 parts:\r\n\r\n* a parent job that will load the recipients and enqueue a child job for each\r\n* a child job that will perform a single email delivery\r\n\r\nDoing this removes the memory leak seen in previous version when delivering emails, and gives greater control over how much work is left to do as each job now does 1 thing and will eliminate the need for long running mail delivery jobs.\r\n\r\nThis PR should be merged on top of #359 \r\n\r\nMultiple worker processes can spread the load of the actual email sending, rather than just tying up 1 process for a long time.\r\n\r\nBoth the parent and child jobs will check that the petition they are delivering information for and will exit if the petition has been updated since the jobs were enqueued.\r\n\r\nTried to keep with the existing structure, could do with further refactoring.\r\n\r\nTested locally delivering 10K emails:\r\n\r\n* Worker Memory levels at ~250MB when enqueuing mailing jobs (previously was unbounded growth)\r\n* Worker Memory evens out at ~270MB when running mailing jobs\r\n* Mailing jobs run at ~7 - ~10 job/s (with local mailhost)\r\n\r\nUpdate:\r\nDeliver email jobs are now idempotent. They can be run multiple times but only send an email once (until the petition is updated)\r\n\r\nThings this PR does not address:\r\n\r\n* there is no check to confirm that if a petition has X signatories, X emails actually got sent.\r\n\r\n/cc @h-lame @oskarpearson @alanth @pixeltrix ","created_at":"2015-07-28T14:34:52Z","updated_at":"2015-07-29T08:48:31Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a546e2c53986e63de87ba4abead85b0ae1fb3bb3","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/alphagov/e-petitions/pulls/360/commits","review_comments_url":"https://api.github.com/repos/alphagov/e-petitions/pulls/360/comments","review_comment_url":"https://api.github.com/repos/alphagov/e-petitions/pulls/comments{/number}","comments_url":"https://api.github.com/repos/alphagov/e-petitions/issues/360/comments","statuses_url":"https://api.github.com/repos/alphagov/e-petitions/statuses/2b317b570ee8a1984cf07c57b77931d7db81b65b","head":{"label":"ansonK:email-send-refactor","ref":"email-send-refactor","sha":"2b317b570ee8a1984cf07c57b77931d7db81b65b","user":{"login":"ansonK","id":722405,"avatar_url":"https://avatars.githubusercontent.com/u/722405?v=3","gravatar_id":"","url":"https://api.github.com/users/ansonK","html_url":"https://github.com/ansonK","followers_url":"https://api.github.com/users/ansonK/followers","following_url":"https://api.github.com/users/ansonK/following{/other_user}","gists_url":"https://api.github.com/users/ansonK/gists{/gist_id}","starred_url":"https://api.github.com/users/ansonK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ansonK/subscriptions","organizations_url":"https://api.github.com/users/ansonK/orgs","repos_url":"https://api.github.com/users/ansonK/repos","events_url":"https://api.github.com/users/ansonK/events{/privacy}","received_events_url":"https://api.github.com/users/ansonK/received_events","type":"User","site_admin":false},"repo":{"id":39837032,"name":"e-petitions","full_name":"ansonK/e-petitions","owner":{"login":"ansonK","id":722405,"avatar_url":"https://avatars.githubusercontent.com/u/722405?v=3","gravatar_id":"","url":"https://api.github.com/users/ansonK","html_url":"https://github.com/ansonK","followers_url":"https://api.github.com/users/ansonK/followers","following_url":"https://api.github.com/users/ansonK/following{/other_user}","gists_url":"https://api.github.com/users/ansonK/gists{/gist_id}","starred_url":"https://api.github.com/users/ansonK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ansonK/subscriptions","organizations_url":"https://api.github.com/users/ansonK/orgs","repos_url":"https://api.github.com/users/ansonK/repos","events_url":"https://api.github.com/users/ansonK/events{/privacy}","received_events_url":"https://api.github.com/users/ansonK/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ansonK/e-petitions","description":"This is the code base for the UK Government's e-petitions service (http://epetitions.direct.gov.uk)","fork":true,"url":"https://api.github.com/repos/ansonK/e-petitions","forks_url":"https://api.github.com/repos/ansonK/e-petitions/forks","keys_url":"https://api.github.com/repos/ansonK/e-petitions/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ansonK/e-petitions/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ansonK/e-petitions/teams","hooks_url":"https://api.github.com/repos/ansonK/e-petitions/hooks","issue_events_url":"https://api.github.com/repos/ansonK/e-petitions/issues/events{/number}","events_url":"https://api.github.com/repos/ansonK/e-petitions/events","assignees_url":"https://api.github.com/repos/ansonK/e-petitions/assignees{/user}","branches_url":"https://api.github.com/repos/ansonK/e-petitions/branches{/branch}","tags_url":"https://api.github.com/repos/ansonK/e-petitions/tags","blobs_url":"https://api.github.com/repos/ansonK/e-petitions/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ansonK/e-petitions/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ansonK/e-petitions/git/refs{/sha}","trees_url":"https://api.github.com/repos/ansonK/e-petitions/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ansonK/e-petitions/statuses/{sha}","languages_url":"https://api.github.com/repos/ansonK/e-petitions/languages","stargazers_url":"https://api.github.com/repos/ansonK/e-petitions/stargazers","contributors_url":"https://api.github.com/repos/ansonK/e-petitions/contributors","subscribers_url":"https://api.github.com/repos/ansonK/e-petitions/subscribers","subscription_url":"https://api.github.com/repos/ansonK/e-petitions/subscription","commits_url":"https://api.github.com/repos/ansonK/e-petitions/commits{/sha}","git_commits_url":"https://api.github.com/repos/ansonK/e-petitions/git/commits{/sha}","comments_url":"https://api.github.com/repos/ansonK/e-petitions/comments{/number}","issue_comment_url":"https://api.github.com/repos/ansonK/e-petitions/issues/comments{/number}","contents_url":"https://api.github.com/repos/ansonK/e-petitions/contents/{+path}","compare_url":"https://api.github.com/repos/ansonK/e-petitions/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ansonK/e-petitions/merges","archive_url":"https://api.github.com/repos/ansonK/e-petitions/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ansonK/e-petitions/downloads","issues_url":"https://api.github.com/repos/ansonK/e-petitions/issues{/number}","pulls_url":"https://api.github.com/repos/ansonK/e-petitions/pulls{/number}","milestones_url":"https://api.github.com/repos/ansonK/e-petitions/milestones{/number}","notifications_url":"https://api.github.com/repos/ansonK/e-petitions/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ansonK/e-petitions/labels{/name}","releases_url":"https://api.github.com/repos/ansonK/e-petitions/releases{/id}","created_at":"2015-07-28T13:58:12Z","updated_at":"2015-07-28T13:58:13Z","pushed_at":"2015-07-28T16:12:28Z","git_url":"git://github.com/ansonK/e-petitions.git","ssh_url":"git@github.com:ansonK/e-petitions.git","clone_url":"https://github.com/ansonK/e-petitions.git","svn_url":"https://github.com/ansonK/e-petitions","homepage":"http://epetitions.direct.gov.uk","size":9014,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"alphagov:master","ref":"master","sha":"d3eaf2c9a150132622ac463b4843bcd577090c55","user":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":4434819,"name":"e-petitions","full_name":"alphagov/e-petitions","owner":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/alphagov/e-petitions","description":"This is the code base for the UK Government's e-petitions service (http://epetitions.direct.gov.uk)","fork":false,"url":"https://api.github.com/repos/alphagov/e-petitions","forks_url":"https://api.github.com/repos/alphagov/e-petitions/forks","keys_url":"https://api.github.com/repos/alphagov/e-petitions/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/e-petitions/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/e-petitions/teams","hooks_url":"https://api.github.com/repos/alphagov/e-petitions/hooks","issue_events_url":"https://api.github.com/repos/alphagov/e-petitions/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/e-petitions/events","assignees_url":"https://api.github.com/repos/alphagov/e-petitions/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/e-petitions/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/e-petitions/tags","blobs_url":"https://api.github.com/repos/alphagov/e-petitions/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/e-petitions/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/e-petitions/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/e-petitions/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/e-petitions/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/e-petitions/languages","stargazers_url":"https://api.github.com/repos/alphagov/e-petitions/stargazers","contributors_url":"https://api.github.com/repos/alphagov/e-petitions/contributors","subscribers_url":"https://api.github.com/repos/alphagov/e-petitions/subscribers","subscription_url":"https://api.github.com/repos/alphagov/e-petitions/subscription","commits_url":"https://api.github.com/repos/alphagov/e-petitions/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/e-petitions/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/e-petitions/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/e-petitions/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/e-petitions/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/e-petitions/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/e-petitions/merges","archive_url":"https://api.github.com/repos/alphagov/e-petitions/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/e-petitions/downloads","issues_url":"https://api.github.com/repos/alphagov/e-petitions/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/e-petitions/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/e-petitions/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/e-petitions/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/e-petitions/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/e-petitions/releases{/id}","created_at":"2012-05-24T16:04:02Z","updated_at":"2015-07-21T17:38:14Z","pushed_at":"2015-07-28T16:12:29Z","git_url":"git://github.com/alphagov/e-petitions.git","ssh_url":"git@github.com:alphagov/e-petitions.git","clone_url":"https://github.com/alphagov/e-petitions.git","svn_url":"https://github.com/alphagov/e-petitions","homepage":"http://epetitions.direct.gov.uk","size":9014,"stargazers_count":23,"watchers_count":23,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13,"mirror_url":null,"open_issues_count":9,"forks":13,"open_issues":9,"watchers":23,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/360"},"html":{"href":"https://github.com/alphagov/e-petitions/pull/360"},"issue":{"href":"https://api.github.com/repos/alphagov/e-petitions/issues/360"},"comments":{"href":"https://api.github.com/repos/alphagov/e-petitions/issues/360/comments"},"review_comments":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/360/comments"},"review_comment":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/alphagov/e-petitions/pulls/360/commits"},"statuses":{"href":"https://api.github.com/repos/alphagov/e-petitions/statuses/2b317b570ee8a1984cf07c57b77931d7db81b65b"}}}}