{"action":"created","issue":{"url":"https://api.github.com/repos/evantahler/actionhero/issues/654","labels_url":"https://api.github.com/repos/evantahler/actionhero/issues/654/labels{/name}","comments_url":"https://api.github.com/repos/evantahler/actionhero/issues/654/comments","events_url":"https://api.github.com/repos/evantahler/actionhero/issues/654/events","html_url":"https://github.com/evantahler/actionhero/issues/654","id":75338304,"number":654,"title":"Periodic Task will not enqueue on startup, can't remove it or start it using Grunt","user":{"login":"awinder","id":32653,"avatar_url":"https://avatars.githubusercontent.com/u/32653?v=3","gravatar_id":"","url":"https://api.github.com/users/awinder","html_url":"https://github.com/awinder","followers_url":"https://api.github.com/users/awinder/followers","following_url":"https://api.github.com/users/awinder/following{/other_user}","gists_url":"https://api.github.com/users/awinder/gists{/gist_id}","starred_url":"https://api.github.com/users/awinder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awinder/subscriptions","organizations_url":"https://api.github.com/users/awinder/orgs","repos_url":"https://api.github.com/users/awinder/repos","events_url":"https://api.github.com/users/awinder/events{/privacy}","received_events_url":"https://api.github.com/users/awinder/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-11T19:53:37Z","updated_at":"2015-05-13T13:37:21Z","closed_at":"2015-05-13T13:37:21Z","body":"I have a periodic task that seems to get stuck on unclean shutdowns (i.e., killing the actionhero process and not allowing it to shut down cleanly).  When I bring the service back up, it never gets re-enqueued.  I saw the `grunt` helpers that seemed like they would be able to clear this problem up, but I'm getting the following output attempting to use them:\r\n\r\n```\r\n# grunt stopPeriodicTask:cacheToken\r\nRunning \"stopPeriodicTask:cacheToken\" (stopPeriodicTask) task\r\nremoved 0 instances of cacheToken\r\n\r\nDone, without errors.\r\n# grunt enqueuePeriodicTask:cacheToken\r\nRunning \"enqueuePeriodicTask:cacheToken\" (enqueuePeriodicTask) task\r\ncacheToken not enqueued\r\n```\r\n\r\nNot really sure what to do other than to rename the periodic task, but this has happened a number of times so I'm kinda worried that this might be a not-so-rare occurance"},"comment":{"url":"https://api.github.com/repos/evantahler/actionhero/issues/comments/101665471","html_url":"https://github.com/evantahler/actionhero/issues/654#issuecomment-101665471","issue_url":"https://api.github.com/repos/evantahler/actionhero/issues/654","id":101665471,"user":{"login":"evantahler","id":303226,"avatar_url":"https://avatars.githubusercontent.com/u/303226?v=3","gravatar_id":"","url":"https://api.github.com/users/evantahler","html_url":"https://github.com/evantahler","followers_url":"https://api.github.com/users/evantahler/followers","following_url":"https://api.github.com/users/evantahler/following{/other_user}","gists_url":"https://api.github.com/users/evantahler/gists{/gist_id}","starred_url":"https://api.github.com/users/evantahler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evantahler/subscriptions","organizations_url":"https://api.github.com/users/evantahler/orgs","repos_url":"https://api.github.com/users/evantahler/repos","events_url":"https://api.github.com/users/evantahler/events{/privacy}","received_events_url":"https://api.github.com/users/evantahler/received_events","type":"User","site_admin":false},"created_at":"2015-05-13T13:37:21Z","updated_at":"2015-05-13T13:37:21Z","body":"**tl;dr**: it's important to actually shut down workers properly.\r\n\r\n---\r\n\r\nIn the situation you describe, this is actually the expected behavior, and here's why:\r\n\r\n- When you `kill 9` and actionhero worker, the redis keys which indicated that the periodic tasks was claimed and being worked on are never removed:\r\n\r\n![screen shot 2015-05-13 at 2 24 01 pm](https://cloud.githubusercontent.com/assets/303226/7611337/bd13b3ca-f97b-11e4-99f5-0aff548f033d.png)\r\n\r\nThe periodic tasks use 3 [resque plugins](https://github.com/evantahler/actionhero/blob/master/initializers/tasks.js#L44-L48) `jobLock`, `queueLock`, and `workerLock`, which in turn, prevent more than one instance of that job ever being enqueued, enqueued in the delayed queue, or being worked on... which it totally what you want.  \r\n\r\nNow, with those keys still being in redis, when actionhero starts and checks on the delayed jobs which should be enqueued, it sees an (old) worker already working on your `cacheToken` job, so it thinks all is well and moves on. These resque plugins do have [timeouts](https://github.com/taskrabbit/node-resque/blob/master/lib/plugins/jobLock.js#L68-L75), but waiting that long is probably not a good idea.\r\n\r\nResque is designed not have a heartbeat or any real worker status mechanism, because it assumes single threaded applications/languages like node.  If your task is CPU-intensive, the event loop will be blocked, so it can't \"ping\" redis every so often with a \"I'm still working\" message.  It's up to you to handle failures/errors with your periodic tasks.\r\n\r\nWe recently added 2 methods to node-resque which are applicable here: `queue.cleanOldWorkers(age, callback)` and `queue.forceCleanWorker(workerName, callback)`.  I'm opting *not* to use these methods within actionhero by default, as they are dangerous (you can read the caveats [here](https://github.com/taskrabbit/node-resque#failed-worker-managment) and the related PR for actionhero [here](https://github.com/evantahler/actionhero/pull/655))\r\n\r\nSo the action for you here is to make an initializer that runs at boot to check for these dead workers using your own heuristics (Can you tell they are dead from the start time of the job?  Can you tell they are dead from the host name?).  Once you find those dead workers, you can use `queue.forceCleanWorker(workerName, callback)` (once PR #655 gets merged)\r\n"}}