{"action":"created","issue":{"url":"https://api.github.com/repos/nevans/resque-pool/issues/132","labels_url":"https://api.github.com/repos/nevans/resque-pool/issues/132/labels{/name}","comments_url":"https://api.github.com/repos/nevans/resque-pool/issues/132/comments","events_url":"https://api.github.com/repos/nevans/resque-pool/issues/132/events","html_url":"https://github.com/nevans/resque-pool/pull/132","id":110051752,"number":132,"title":"Options to aid zero-downtime deployment / process supervision","user":{"login":"brasic","id":677224,"avatar_url":"https://avatars.githubusercontent.com/u/677224?v=3","gravatar_id":"","url":"https://api.github.com/users/brasic","html_url":"https://github.com/brasic","followers_url":"https://api.github.com/users/brasic/followers","following_url":"https://api.github.com/users/brasic/following{/other_user}","gists_url":"https://api.github.com/users/brasic/gists{/gist_id}","starred_url":"https://api.github.com/users/brasic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brasic/subscriptions","organizations_url":"https://api.github.com/users/brasic/orgs","repos_url":"https://api.github.com/users/brasic/repos","events_url":"https://api.github.com/users/brasic/events{/privacy}","received_events_url":"https://api.github.com/users/brasic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-06T16:49:16Z","updated_at":"2015-10-06T17:20:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/nevans/resque-pool/pulls/132","html_url":"https://github.com/nevans/resque-pool/pull/132","diff_url":"https://github.com/nevans/resque-pool/pull/132.diff","patch_url":"https://github.com/nevans/resque-pool/pull/132.patch"},"body":"This PR includes the remaining differences in [ShippingEasy's resque-pool fork](https://github.com/ShippingEasy/resque-pool), both of which are optional command line flags that tweak startup behavior to support zero-downtime deployment strategies and running resque-pool inside a process supervisor.\r\n\r\nThe normal approach to deploying new worker code is to signal resque-pool to gracefully shutdown with `QUIT` or `INT`, then start up a new instance.  The problem with that approach is that long-running jobs and slow app startup can lead to a dramatic reduction in worker throughput during a deploy, which in our case is unacceptable.\r\n\r\nThe technique we use to maintain full throughput during deploys is to start a second resque-pool instance with the new code which loads a rails environment and gracefully shuts down the previous instance only when its children are ready to accept work.  This requires two new command line options:\r\n\r\n1) `--no-pidfile`: Launching a second resque-pool instance in daemon mode was not possible because it detects the prior instance by its pidfile and refuses to start.  This option causes the daemon to skip pidfile creation so multiple instances can run at the same time.  Of course, this should only be used if the daemon is run under a supervisor like upstart or systemd that can automatically detect the pids of managed processes.\r\n\r\n1) `--lock FILE`: Even though upstart and systemd can detect the pids of processes they manage, neither copes well with code redeploys, where an entirely new process replaces the existing one.  For this we rely on a approach adapted from [this post](http://orchestrate.io/blog/2014/03/26/tip-seamless-restarts-with-unicorn-and-upstart/) where resque-pool opens a shared filesystem lock at startup on a designated file.  Deploys cause the new process to open an additional shared lock on the same file before the old process exits, meaning that as long as at least one instance is running the file will always be locked in shared mode.  The process supervisor starts resque-pool then immediately attempts to open an exclusive lock on the same file.  This will block until all shared locks are cleared, so the supervisor will only attempt to restart resque-pool when there are truly no instances running.  In this way the shared lock file acts as surrogate pid file, except it can be shared across redeployed processes.\r\n\r\nThis approach is a little unusual but it works well for us - we are able to deploy many times per day under heavy load with zero impact on job throughput or latency.  We would like to switch from our fork to the official version, so I hope these contributions will be useful to others."},"comment":{"url":"https://api.github.com/repos/nevans/resque-pool/issues/comments/145932866","html_url":"https://github.com/nevans/resque-pool/pull/132#issuecomment-145932866","issue_url":"https://api.github.com/repos/nevans/resque-pool/issues/132","id":145932866,"user":{"login":"joshuaflanagan","id":81455,"avatar_url":"https://avatars.githubusercontent.com/u/81455?v=3","gravatar_id":"","url":"https://api.github.com/users/joshuaflanagan","html_url":"https://github.com/joshuaflanagan","followers_url":"https://api.github.com/users/joshuaflanagan/followers","following_url":"https://api.github.com/users/joshuaflanagan/following{/other_user}","gists_url":"https://api.github.com/users/joshuaflanagan/gists{/gist_id}","starred_url":"https://api.github.com/users/joshuaflanagan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshuaflanagan/subscriptions","organizations_url":"https://api.github.com/users/joshuaflanagan/orgs","repos_url":"https://api.github.com/users/joshuaflanagan/repos","events_url":"https://api.github.com/users/joshuaflanagan/events{/privacy}","received_events_url":"https://api.github.com/users/joshuaflanagan/received_events","type":"User","site_admin":false},"created_at":"2015-10-06T17:20:38Z","updated_at":"2015-10-06T17:20:38Z","body":":+1: This code has been running reliably for us for almost a year."}}