{"action":"created","issue":{"url":"https://api.github.com/repos/wikimedia/service-runner/issues/44","labels_url":"https://api.github.com/repos/wikimedia/service-runner/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/wikimedia/service-runner/issues/44/comments","events_url":"https://api.github.com/repos/wikimedia/service-runner/issues/44/events","html_url":"https://github.com/wikimedia/service-runner/pull/44","id":101015823,"number":44,"title":"Implemented heartbeat in service-runner","user":{"login":"Pchelolo","id":814754,"avatar_url":"https://avatars.githubusercontent.com/u/814754?v=3","gravatar_id":"","url":"https://api.github.com/users/Pchelolo","html_url":"https://github.com/Pchelolo","followers_url":"https://api.github.com/users/Pchelolo/followers","following_url":"https://api.github.com/users/Pchelolo/following{/other_user}","gists_url":"https://api.github.com/users/Pchelolo/gists{/gist_id}","starred_url":"https://api.github.com/users/Pchelolo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Pchelolo/subscriptions","organizations_url":"https://api.github.com/users/Pchelolo/orgs","repos_url":"https://api.github.com/users/Pchelolo/repos","events_url":"https://api.github.com/users/Pchelolo/events{/privacy}","received_events_url":"https://api.github.com/users/Pchelolo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-08-14T13:58:11Z","updated_at":"2015-08-14T13:58:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/wikimedia/service-runner/pulls/44","html_url":"https://github.com/wikimedia/service-runner/pull/44","diff_url":"https://github.com/wikimedia/service-runner/pull/44.diff","patch_url":"https://github.com/wikimedia/service-runner/pull/44.patch"},"body":"The ticket opens multiple questions, but this PR tackles the solutions with heartbeat from workers, which is tracked by master, and if it stops, the an attempt to make a graceful shutdown is made, if that fails, woke is killed and restarted. \r\n\r\nI've been feeling myself as a serial killer today, killing and breaking workers with different kinds of weapons. The biggest problem is CPU-bound loops. If a worker got in some kind of `while(true)`, lots of weird things start happening:\r\n- Obviously, before the fix we couldn't detect it and had it running forever.\r\n- Killing it with `worker.kill('SIGKILL')` doesn't work, because `worker.kill` doesn't send a signal right away, but a disconnect first, which can't finish for a worker that's in a loop.\r\n- Killing such worker with console `kill -9 <pid>` crashes the whole server!!! (including master). The problem is this bug in node: https://github.com/joyent/node/issues/9409 Master's notion of workers is updated on `disconnect` message and checked on `exit`. This was fixed by a dirty hack which replaces node's listeners on a worker, but the hack works and it looks pretty stable.\r\n\r\nI've run the service with the fix for quite some time, performed some benchmarking, killing workers in the process, and everything looks stable. What are your opinions on this? Do we want a dirty hack? \r\n\r\nBug: https://phabricator.wikimedia.org/T90362"},"comment":{"url":"https://api.github.com/repos/wikimedia/service-runner/issues/comments/131113413","html_url":"https://github.com/wikimedia/service-runner/pull/44#issuecomment-131113413","issue_url":"https://api.github.com/repos/wikimedia/service-runner/issues/44","id":131113413,"user":{"login":"Pchelolo","id":814754,"avatar_url":"https://avatars.githubusercontent.com/u/814754?v=3","gravatar_id":"","url":"https://api.github.com/users/Pchelolo","html_url":"https://github.com/Pchelolo","followers_url":"https://api.github.com/users/Pchelolo/followers","following_url":"https://api.github.com/users/Pchelolo/following{/other_user}","gists_url":"https://api.github.com/users/Pchelolo/gists{/gist_id}","starred_url":"https://api.github.com/users/Pchelolo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Pchelolo/subscriptions","organizations_url":"https://api.github.com/users/Pchelolo/orgs","repos_url":"https://api.github.com/users/Pchelolo/repos","events_url":"https://api.github.com/users/Pchelolo/events{/privacy}","received_events_url":"https://api.github.com/users/Pchelolo/received_events","type":"User","site_admin":false},"created_at":"2015-08-14T13:58:26Z","updated_at":"2015-08-14T13:58:26Z","body":"/cc @gwicke @d00rman @eevans "}}