{"action":"created","issue":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/2474","labels_url":"https://api.github.com/repos/mperham/sidekiq/issues/2474/labels{/name}","comments_url":"https://api.github.com/repos/mperham/sidekiq/issues/2474/comments","events_url":"https://api.github.com/repos/mperham/sidekiq/issues/2474/events","html_url":"https://github.com/mperham/sidekiq/issues/2474","id":99032734,"number":2474,"title":"Missing jobs in batch ","user":{"login":"Saicheg","id":624999,"avatar_url":"https://avatars.githubusercontent.com/u/624999?v=3","gravatar_id":"","url":"https://api.github.com/users/Saicheg","html_url":"https://github.com/Saicheg","followers_url":"https://api.github.com/users/Saicheg/followers","following_url":"https://api.github.com/users/Saicheg/following{/other_user}","gists_url":"https://api.github.com/users/Saicheg/gists{/gist_id}","starred_url":"https://api.github.com/users/Saicheg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Saicheg/subscriptions","organizations_url":"https://api.github.com/users/Saicheg/orgs","repos_url":"https://api.github.com/users/Saicheg/repos","events_url":"https://api.github.com/users/Saicheg/events{/privacy}","received_events_url":"https://api.github.com/users/Saicheg/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mperham/sidekiq/labels/pro","name":"pro","color":"e10c02"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-08-04T18:41:27Z","updated_at":"2015-09-14T17:07:21Z","closed_at":null,"body":"As i described on gitter. I realized a feature which uses sidekiq batches under the hood. Everythere were pretty fine on development and test. But in production after i missed some jobs on the 1st worker launch. I mean COMPLETE missed. Can't find them anywhere, in queue, failures, process.\r\n\r\nHere is screenshot of how it looks: https://yadi.sk/i/zjnHLByriGMWz\r\n6 hours after after couple full sidekiq restarts it is still there: https://yadi.sk/i/MVr1C7gSiGf9h\r\n\r\nMy sidekiq config:\r\n\r\n```\r\nrequire 'sidekiq-pro'\r\n\r\nraw_yaml = YAML.load(File.read(File.join(File.expand_path('../../config/redis.yml', \"config/redis.yml\"))))\r\nenv = ENV['RAILS_ENV'] || 'development'\r\nredis_config = raw_yaml[env]\r\n\r\nsidekiq_config = YAML.load(File.read(File.join(File.expand_path('../../config/sidekiq.yml', \"config/sidekiq.yml\"))))\r\n\r\nSidekiq.configure_server do |config|\r\n  config.reliable_fetch!\r\n  config.reliable_scheduler!\r\n\r\n  config.redis = {\r\n      url: \"redis://#{redis_config['host']}:#{redis_config['port']}/\",\r\n      namespace: 'sidekiq_clickcast'\r\n  }\r\n  ActiveRecord::Base.configurations[env]['pool'] = sidekiq_config[:concurrency].to_i + 2\r\n  ActiveRecord::Base.establish_connection\r\nend\r\n\r\nSidekiq.configure_client do |config|\r\n  config.redis = {\r\n      url: \"redis://#{redis_config['host']}:#{redis_config['port']}/\",\r\n      namespace: 'sidekiq_clickcast'\r\n  }\r\nend\r\n\r\nSidekiq::Client.reliable_push! if env == 'staging' || env == 'production'\r\n```\r\n\r\nGems:\r\n```\r\ngem 'sidekiq', '~> 3.4.2'\r\ngem 'sidekiq-client-cli', '~> 0.1.7'\r\ngem 'sidekiq-failures', github: 'mhfs/sidekiq-failures'                                                                                                       gem 'sidekiq-unique-jobs', '~> 3.0.2'\r\ngem 'sidekiq-limit_fetch', '~> 2.2.6'\r\nsource <STRIPPED> do\r\n  gem 'sidekiq-pro'             \r\nend \r\n```\r\n\r\nI can't reproduce it anymore. All other launches went fine.                                                                                                                        "},"comment":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/comments/140145966","html_url":"https://github.com/mperham/sidekiq/issues/2474#issuecomment-140145966","issue_url":"https://api.github.com/repos/mperham/sidekiq/issues/2474","id":140145966,"user":{"login":"pacoguzman","id":10926,"avatar_url":"https://avatars.githubusercontent.com/u/10926?v=3","gravatar_id":"","url":"https://api.github.com/users/pacoguzman","html_url":"https://github.com/pacoguzman","followers_url":"https://api.github.com/users/pacoguzman/followers","following_url":"https://api.github.com/users/pacoguzman/following{/other_user}","gists_url":"https://api.github.com/users/pacoguzman/gists{/gist_id}","starred_url":"https://api.github.com/users/pacoguzman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pacoguzman/subscriptions","organizations_url":"https://api.github.com/users/pacoguzman/orgs","repos_url":"https://api.github.com/users/pacoguzman/repos","events_url":"https://api.github.com/users/pacoguzman/events{/privacy}","received_events_url":"https://api.github.com/users/pacoguzman/received_events","type":"User","site_admin":false},"created_at":"2015-09-14T17:07:21Z","updated_at":"2015-09-14T17:07:21Z","body":"Thanks for the info.\n\nIf I understood it properly step one is not critical you can always\nreconcile the jobs of those instances that didn't stop properly. Is that\nright? Or I'm missing something.\n\nBy reconcile I'm thinking in moving to the corresponding queue, not the per\nprocess queue, the jobs that still are on the process queues between\nexecution of the \"script\".\n\nWe thought on some expiration mechanism but that seems a lot easier and\nprobably less resource expensive.\n\nThanks again\nOn Sep 14, 2015 6:42 PM, \"Jonathan Hyman\" <notifications@github.com> wrote:\n\n> When we autoscale down, we first run a script which gracefully stops all\n> the Sidekiq workers, and returns an error if they do not stop after the\n> timeout period, and that aborts the auto-scale-down.\n>\n> As for pending jobs in internal queues, we also wrote our own monitoring\n> that does a scan_each with the pattern queue:*_*_* , saves everything it\n> finds, and diffs every half hour to see if we have any jobs that are\n> \"stuck\" (where stuck could be due to a frozen process or a machine that\n> died that had jobs in an internal queue). We then manually reconcile, as it\n> is an uncommon occurrence.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/mperham/sidekiq/issues/2474#issuecomment-140140048>.\n>\n"}}