{"action":"created","issue":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/2531","labels_url":"https://api.github.com/repos/mperham/sidekiq/issues/2531/labels{/name}","comments_url":"https://api.github.com/repos/mperham/sidekiq/issues/2531/comments","events_url":"https://api.github.com/repos/mperham/sidekiq/issues/2531/events","html_url":"https://github.com/mperham/sidekiq/issues/2531","id":104583305,"number":2531,"title":"Sidekiq can lose jobs even with reliable fetch","user":{"login":"sanjayprabhu","id":75855,"avatar_url":"https://avatars.githubusercontent.com/u/75855?v=3","gravatar_id":"","url":"https://api.github.com/users/sanjayprabhu","html_url":"https://github.com/sanjayprabhu","followers_url":"https://api.github.com/users/sanjayprabhu/followers","following_url":"https://api.github.com/users/sanjayprabhu/following{/other_user}","gists_url":"https://api.github.com/users/sanjayprabhu/gists{/gist_id}","starred_url":"https://api.github.com/users/sanjayprabhu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanjayprabhu/subscriptions","organizations_url":"https://api.github.com/users/sanjayprabhu/orgs","repos_url":"https://api.github.com/users/sanjayprabhu/repos","events_url":"https://api.github.com/users/sanjayprabhu/events{/privacy}","received_events_url":"https://api.github.com/users/sanjayprabhu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-02T22:17:54Z","updated_at":"2015-09-02T22:27:11Z","closed_at":null,"body":"I'm on sidekiq version `2.17.8` (and sidekiq-pro `1.5.1`) and investigating an issue where sidekiq is not performing some jobs even with reliable queueing enabled. \r\n\r\nCurrently, looks like this can happen if there was an exception in any of the middleware in the stack before it hits the `RetryJobs` middleware. Or, say, an exception in the `RetryJobs` middleware anytime before it queues up the retry (less likely).\r\n\r\nThis is because in [this block of code](https://github.com/mperham/sidekiq/blob/v2.17.8/lib/sidekiq/processor.rb#L40-L62):\r\n\r\n```ruby\r\n        ack = true\r\n        begin\r\n          msg = Sidekiq.load_json(msgstr)\r\n          klass  = msg['class'].constantize\r\n          worker = klass.new\r\n          worker.jid = msg['jid']\r\n\r\n          stats(worker, msg, queue) do\r\n            Sidekiq.server_middleware.invoke(worker, msg, queue) do\r\n              worker.perform(*cloned(msg['args']))\r\n            end\r\n          end\r\n        rescue Sidekiq::Shutdown\r\n          # Had to force kill this job because it didn't finish\r\n          # within the timeout.  Don't acknowledge the work since\r\n          # we didn't properly finish it.\r\n          ack = false\r\n        rescue Exception => ex\r\n          handle_exception(ex, msg || { :message => msgstr })\r\n          raise\r\n        ensure\r\n          work.acknowledge if ack\r\n        end\r\n```\r\n\r\nIt ack's the job even though the middleware may not have executed successfully. You can reproduce this locally by running sidekiq within foreman, sending it a lot of jobs and `ctrl-c`ing it. [This line](https://github.com/mperham/sidekiq/blob/v2.17.8/lib/sidekiq/middleware/server/logging.rb#L10) fails with `Broken pipe @ io_write - <STDERR>` and you lose the job (`perform` was never called on the worker, `RetryJobs` was never reached, so it doesn't try to retry but we still ack the work so nobody else is going to work on it). My [test worker](https://gist.github.com/sanjayprabhu/350d952f47c5a71fdae2) if you're interested.\r\n\r\nThoughts on how to fix this? Maybe start with `ack=false` and do something like\r\n\r\n```ruby\r\n            Sidekiq.server_middleware.invoke(worker, msg, queue) do\r\n              ack = true # we attempted to perform the job\r\n              worker.perform(*cloned(msg['args']))\r\n            end\r\n            ack = true # middleware invoked successfully (may or may not have decided to perform the job)\r\n```\r\n\r\nI can open a PR for that if you think that sounds reasonable.\r\n\r\nThe real kicker is, this codepath is not hit in production for us (`RetryJobs` is the first middleware in our chain), so there's at least one more bug which is causing us to lose jobs. I'm continuing to investigate that."},"comment":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/comments/137262912","html_url":"https://github.com/mperham/sidekiq/issues/2531#issuecomment-137262912","issue_url":"https://api.github.com/repos/mperham/sidekiq/issues/2531","id":137262912,"user":{"login":"ryansch","id":146013,"avatar_url":"https://avatars.githubusercontent.com/u/146013?v=3","gravatar_id":"","url":"https://api.github.com/users/ryansch","html_url":"https://github.com/ryansch","followers_url":"https://api.github.com/users/ryansch/followers","following_url":"https://api.github.com/users/ryansch/following{/other_user}","gists_url":"https://api.github.com/users/ryansch/gists{/gist_id}","starred_url":"https://api.github.com/users/ryansch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryansch/subscriptions","organizations_url":"https://api.github.com/users/ryansch/orgs","repos_url":"https://api.github.com/users/ryansch/repos","events_url":"https://api.github.com/users/ryansch/events{/privacy}","received_events_url":"https://api.github.com/users/ryansch/received_events","type":"User","site_admin":false},"created_at":"2015-09-02T22:27:11Z","updated_at":"2015-09-02T22:27:11Z","body":"Interestingly, that method is exactly the same on `v3.5.0` so this may still be an issue.\r\n\r\n@sanjayprabhu Can you reproduce this on the newest releases of sidekiq and sidekiq-pro?"}}