{"action":"closed","issue":{"url":"https://api.github.com/repos/nvie/rq/issues/584","labels_url":"https://api.github.com/repos/nvie/rq/issues/584/labels{/name}","comments_url":"https://api.github.com/repos/nvie/rq/issues/584/comments","events_url":"https://api.github.com/repos/nvie/rq/issues/584/events","html_url":"https://github.com/nvie/rq/issues/584","id":109359855,"number":584,"title":"warm shutdown on heroku and long jobs","user":{"login":"samuelcolvin","id":4039449,"avatar_url":"https://avatars.githubusercontent.com/u/4039449?v=3","gravatar_id":"","url":"https://api.github.com/users/samuelcolvin","html_url":"https://github.com/samuelcolvin","followers_url":"https://api.github.com/users/samuelcolvin/followers","following_url":"https://api.github.com/users/samuelcolvin/following{/other_user}","gists_url":"https://api.github.com/users/samuelcolvin/gists{/gist_id}","starred_url":"https://api.github.com/users/samuelcolvin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samuelcolvin/subscriptions","organizations_url":"https://api.github.com/users/samuelcolvin/orgs","repos_url":"https://api.github.com/users/samuelcolvin/repos","events_url":"https://api.github.com/users/samuelcolvin/events{/privacy}","received_events_url":"https://api.github.com/users/samuelcolvin/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-01T18:47:58Z","updated_at":"2015-11-08T11:51:39Z","closed_at":"2015-11-08T11:51:39Z","body":"We have some jobs which run on a heroku worker and take quite a long time lets say 1-10 minutes.\r\n\r\nAs per [this](http://support.hirefire.io/discussions/problems/31480-valid-jobs-being-moved-to-failed-in-rq-on-warm-shutdown) or due to heroku restarting dynos these jobs occasionally recieve the `SIGTERM` signal and then shutdown.\r\n\r\nSince the jobs are unlikely to end in the 10 second grace period Heroku give before killing the dyno the right thing to do here would be to raise an Exception and let the job fail. Then our standard code for recording and dealing with errors would work.\r\n\r\nI've tried adding a handler to `SIGTERM` to do this however it's not being called because of [_install_signal_handlers](https://github.com/nvie/rq/blob/bb4336f612910f33ed72359a3f8b690b4acf8a4a/rq/worker.py#L321-L366). Because the jobs are eventually killed by the worker shutting down we're not getting an exception raised or sent to sentry, similarly the job doesn't seem to appear in the failed queue until it's exceed it's timeout (or perhaps \"cleanup\" deals with it, I don't know).\r\n\r\nWhat's the best solution here?\r\n\r\nAll I can think of here is to create a custom worker and overwrite `_install_signal_handlers`. If that's the correct solution can I suggest defining `request_force_stop` and `request_stop` outside `_install_signal_handlers` so less code has to be duplicated?"}}