{"action":"created","issue":{"url":"https://api.github.com/repos/Automattic/kue/issues/716","labels_url":"https://api.github.com/repos/Automattic/kue/issues/716/labels{/name}","comments_url":"https://api.github.com/repos/Automattic/kue/issues/716/comments","events_url":"https://api.github.com/repos/Automattic/kue/issues/716/events","html_url":"https://github.com/Automattic/kue/issues/716","id":108104311,"number":716,"title":"Does kue fail after a few million tasks ?","user":{"login":"AaronO","id":949223,"avatar_url":"https://avatars.githubusercontent.com/u/949223?v=3","gravatar_id":"","url":"https://api.github.com/users/AaronO","html_url":"https://github.com/AaronO","followers_url":"https://api.github.com/users/AaronO/followers","following_url":"https://api.github.com/users/AaronO/following{/other_user}","gists_url":"https://api.github.com/users/AaronO/gists{/gist_id}","starred_url":"https://api.github.com/users/AaronO/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AaronO/subscriptions","organizations_url":"https://api.github.com/users/AaronO/orgs","repos_url":"https://api.github.com/users/AaronO/repos","events_url":"https://api.github.com/users/AaronO/events{/privacy}","received_events_url":"https://api.github.com/users/AaronO/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-24T10:40:52Z","updated_at":"2015-09-24T12:25:21Z","closed_at":null,"body":"Every time we reach about ~10,000,000 tasks, `kue` / `redis` seems to get stuck. It can add new tasks to `redis` but `kue` stops consuming tasks so they just pile up ...\r\n\r\nI was wondering if anyone else has experienced this issue with `kue` in production and has an idea why `kue` would fail ?"},"comment":{"url":"https://api.github.com/repos/Automattic/kue/issues/comments/142913922","html_url":"https://github.com/Automattic/kue/issues/716#issuecomment-142913922","issue_url":"https://api.github.com/repos/Automattic/kue/issues/716","id":142913922,"user":{"login":"behrad","id":2151889,"avatar_url":"https://avatars.githubusercontent.com/u/2151889?v=3","gravatar_id":"","url":"https://api.github.com/users/behrad","html_url":"https://github.com/behrad","followers_url":"https://api.github.com/users/behrad/followers","following_url":"https://api.github.com/users/behrad/following{/other_user}","gists_url":"https://api.github.com/users/behrad/gists{/gist_id}","starred_url":"https://api.github.com/users/behrad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behrad/subscriptions","organizations_url":"https://api.github.com/users/behrad/orgs","repos_url":"https://api.github.com/users/behrad/repos","events_url":"https://api.github.com/users/behrad/events{/privacy}","received_events_url":"https://api.github.com/users/behrad/received_events","type":"User","site_admin":false},"created_at":"2015-09-24T12:25:21Z","updated_at":"2015-09-24T12:25:21Z","body":"I've a system processing around `500M < X < 1Billion`  jobs now, processing 2M jobs/day.\r\nIf you search issues you find issues with current version on different environments => you should properly gaurd your worker's process method so that node.js process crashes, uncaught exceptions don't  take up job concurrency over time (ttl can be used for this also)\r\n\r\nPlease let me know details if the above didn't help you and you have an specific problem using Kue."}}