{"action":"created","issue":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/2303","labels_url":"https://api.github.com/repos/mperham/sidekiq/issues/2303/labels{/name}","comments_url":"https://api.github.com/repos/mperham/sidekiq/issues/2303/comments","events_url":"https://api.github.com/repos/mperham/sidekiq/issues/2303/events","html_url":"https://github.com/mperham/sidekiq/pull/2303","id":69260596,"number":2303,"title":"Strip retry exceptions when a job is requeued","user":{"login":"zanker","id":323517,"avatar_url":"https://avatars.githubusercontent.com/u/323517?v=3","gravatar_id":"","url":"https://api.github.com/users/zanker","html_url":"https://github.com/zanker","followers_url":"https://api.github.com/users/zanker/followers","following_url":"https://api.github.com/users/zanker/following{/other_user}","gists_url":"https://api.github.com/users/zanker/gists{/gist_id}","starred_url":"https://api.github.com/users/zanker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zanker/subscriptions","organizations_url":"https://api.github.com/users/zanker/orgs","repos_url":"https://api.github.com/users/zanker/repos","events_url":"https://api.github.com/users/zanker/events{/privacy}","received_events_url":"https://api.github.com/users/zanker/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-04-18T03:16:57Z","updated_at":"2015-04-20T16:24:07Z","closed_at":"2015-04-18T10:33:53Z","pull_request":{"url":"https://api.github.com/repos/mperham/sidekiq/pulls/2303","html_url":"https://github.com/mperham/sidekiq/pull/2303","diff_url":"https://github.com/mperham/sidekiq/pull/2303.diff","patch_url":"https://github.com/mperham/sidekiq/pull/2303.patch"},"body":"Thoughts? Will add tests based on what you think. I tried to do a client middleware to make it consistent with how Sidekiq works, rather than just patching it into `normalize_items`.\r\n\r\nIf you enable backtrace logging, whenever a job fails it logs a trace. In a Rails app, it can be pretty big. It can effectively double/triple the payload size. As best I can tell, that persists even when the job is requeued, so you can end up in a situation like this (pseudonumbers):\r\n\r\n100 jobs - 100 bytes used\r\n5 jobs run, they all fail and go into the retry queue, 105 bytes used (5 more for the failure log)\r\n5 jobs are requeued\r\nanother 10 jobs are run, none of them are the original failed jobs, they fail as well, 115 bytes are now used\r\n\r\nRepeat until we run out of memory or the jobs process.\r\n\r\nBecause we add with `lpush` and remove with `brpop`, these retries are going to be processed last. Normally, this isn't a problem. If you say, queue 50,000 jobs and your DB goes down, network is slow/timing out/etc, you can end up having your memory balloon significantly without adding new jobs.\r\n\r\nGiven `backtrace` is disabled by default, this isn't exactly a critical issue since you can also limit how many records you store. I think it's better to not store a lot of redundant data though if it can be avoided.\r\n\r\n \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/comments/94499453","html_url":"https://github.com/mperham/sidekiq/pull/2303#issuecomment-94499453","issue_url":"https://api.github.com/repos/mperham/sidekiq/issues/2303","id":94499453,"user":{"login":"zanker","id":323517,"avatar_url":"https://avatars.githubusercontent.com/u/323517?v=3","gravatar_id":"","url":"https://api.github.com/users/zanker","html_url":"https://github.com/zanker","followers_url":"https://api.github.com/users/zanker/followers","following_url":"https://api.github.com/users/zanker/following{/other_user}","gists_url":"https://api.github.com/users/zanker/gists{/gist_id}","starred_url":"https://api.github.com/users/zanker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zanker/subscriptions","organizations_url":"https://api.github.com/users/zanker/orgs","repos_url":"https://api.github.com/users/zanker/repos","events_url":"https://api.github.com/users/zanker/events{/privacy}","received_events_url":"https://api.github.com/users/zanker/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T16:24:07Z","updated_at":"2015-04-20T16:24:07Z","body":"Fair enough. A warning on the wiki page would be helpful though."}}