{"action":"created","issue":{"url":"https://api.github.com/repos/OptimalBits/bull/issues/180","labels_url":"https://api.github.com/repos/OptimalBits/bull/issues/180/labels{/name}","comments_url":"https://api.github.com/repos/OptimalBits/bull/issues/180/comments","events_url":"https://api.github.com/repos/OptimalBits/bull/issues/180/events","html_url":"https://github.com/OptimalBits/bull/issues/180","id":107780128,"number":180,"title":"Jobs getting stuck under small load","user":{"login":"evanhuang8","id":1572593,"avatar_url":"https://avatars.githubusercontent.com/u/1572593?v=3","gravatar_id":"","url":"https://api.github.com/users/evanhuang8","html_url":"https://github.com/evanhuang8","followers_url":"https://api.github.com/users/evanhuang8/followers","following_url":"https://api.github.com/users/evanhuang8/following{/other_user}","gists_url":"https://api.github.com/users/evanhuang8/gists{/gist_id}","starred_url":"https://api.github.com/users/evanhuang8/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanhuang8/subscriptions","organizations_url":"https://api.github.com/users/evanhuang8/orgs","repos_url":"https://api.github.com/users/evanhuang8/repos","events_url":"https://api.github.com/users/evanhuang8/events{/privacy}","received_events_url":"https://api.github.com/users/evanhuang8/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-22T19:04:42Z","updated_at":"2015-09-22T19:29:56Z","closed_at":null,"body":"I have been very interested in replacing `kue` with `bull` and I have been doing some testing, however, I keep running into the problem of jobs getting stuck.\r\n\r\n### Scripts\r\n\r\nSo my simply setup has two scripts, one is `scheduler` and the other one is `worker`\r\n\r\n`scheduler`:\r\n\r\n```coffeescript\r\n#!/usr/bin/env coffee\r\n\r\nfs = require 'fs'\r\nbull = require 'bull'\r\nmatador = require 'bull-ui/app'\r\nco = require 'co'\r\nQ = require 'q'\r\n\r\ncredentials = JSON.parse fs.readFileSync './credentials.json', 'utf8'\r\n\r\nredisHost = credentials.redisHost\r\nredisPort = credentials.redisPort\r\nredisPass = credentials.redisPass\r\nredisDBNumber = credentials.redisDBNumber\r\n\r\nqueue = bull 'test', redisPort, redisHost\r\n\r\napp = matador\r\n  redis:\r\n    host: redisHost\r\n    port: redisPort\r\n    password: redisPass\r\n\r\nsleep = (duration) ->\r\n  deferred = Q.defer()\r\n  setTimeout ->\r\n    deferred.resolve()\r\n    return\r\n  , duration\r\n  return deferred.promise\r\n\r\nco ->\r\n\r\n  for i in [0...100]\r\n    job = yield queue.add {}, \r\n      delay: 10 * 1000\r\n    console.log \"Job #{job.jobId} scheduled\"\r\n\r\n  yield sleep 1000\r\n\r\n  for i in [0...10]\r\n    for j in [0...10]\r\n      job = yield queue.add {}, \r\n        delay: 8 * 1000\r\n      console.log \"Job #{job.jobId} scheduled\"\r\n    yield sleep 1000\r\n\r\n  return\r\n\r\n.catch (err) ->\r\n\r\n  console.log err\r\n\r\n  return\r\n```\r\n\r\nUsing a bit of ES6 generator logic here, so bare with me.\r\n\r\n`worker`:\r\n\r\n```coffeescript\r\n#!/usr/bin/env coffee\r\n\r\nfs = require 'fs'\r\nbull = require 'bull'\r\nco = require 'co'\r\nQ = require 'q'\r\n\r\ncredentials = JSON.parse fs.readFileSync './credentials.json', 'utf8'\r\n\r\nredisHost = credentials.redisHost\r\nredisPort = credentials.redisPort\r\nredisPass = credentials.redisPass\r\nredisDBNumber = credentials.redisDBNumber\r\n\r\nqueue = bull 'test', redisPort, redisHost\r\n\r\nqueue.process (job, done) ->\r\n  console.log \"Job #{job.jobId} is processed!\"\r\n  done()\r\n  return\r\n\r\nqueue.on 'error', (err) ->\r\n  console.log err\r\n  return\r\n```\r\n\r\n### Test runs\r\n\r\nI am using iojs v2.5.0, tested with both redis 2.8.19 and 3.0.3. I flush redis before every run.\r\n\r\nSo when I run the scheduler first, and wait till all the jobs are scheduled, and then run the worker, then all jobs will go to completion.\r\n\r\nIf I remove the delay settings from the jobs, then everything seems fine - all jobs are processed.\r\n\r\nWhen I run the worker and the scheduler at the same time - in other words, let the worker process as the scheduler goes, only a fraction of the jobs will be processed (ranging between 5 - 50 jobs out of the 200 jobs scheduled). Once the processing stops, no matter how long I wait, the jobs are stuck in the delayed state.\r\n\r\nI suspect there is something going on with how the delayed jobs are setup, any insights? "},"comment":{"url":"https://api.github.com/repos/OptimalBits/bull/issues/comments/142393010","html_url":"https://github.com/OptimalBits/bull/issues/180#issuecomment-142393010","issue_url":"https://api.github.com/repos/OptimalBits/bull/issues/180","id":142393010,"user":{"login":"evanhuang8","id":1572593,"avatar_url":"https://avatars.githubusercontent.com/u/1572593?v=3","gravatar_id":"","url":"https://api.github.com/users/evanhuang8","html_url":"https://github.com/evanhuang8","followers_url":"https://api.github.com/users/evanhuang8/followers","following_url":"https://api.github.com/users/evanhuang8/following{/other_user}","gists_url":"https://api.github.com/users/evanhuang8/gists{/gist_id}","starred_url":"https://api.github.com/users/evanhuang8/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanhuang8/subscriptions","organizations_url":"https://api.github.com/users/evanhuang8/orgs","repos_url":"https://api.github.com/users/evanhuang8/repos","events_url":"https://api.github.com/users/evanhuang8/events{/privacy}","received_events_url":"https://api.github.com/users/evanhuang8/received_events","type":"User","site_admin":false},"created_at":"2015-09-22T19:29:56Z","updated_at":"2015-09-22T19:29:56Z","body":"Yep, here you go:\r\n\r\nhttps://gist.github.com/evanhuang8/e85f76e486eaeefc9e87\r\n\r\nOn a side note, is the option to use a different db number on redis going to be supported any time soon?"}}