{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1155","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1155/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1155/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1155/events","html_url":"https://github.com/hashicorp/consul/issues/1155","id":99074174,"number":1155,"title":"consul lock may spawn the watched process at shutdown","user":{"login":"mfischer-zd","id":984199,"avatar_url":"https://avatars.githubusercontent.com/u/984199?v=3","gravatar_id":"","url":"https://api.github.com/users/mfischer-zd","html_url":"https://github.com/mfischer-zd","followers_url":"https://api.github.com/users/mfischer-zd/followers","following_url":"https://api.github.com/users/mfischer-zd/following{/other_user}","gists_url":"https://api.github.com/users/mfischer-zd/gists{/gist_id}","starred_url":"https://api.github.com/users/mfischer-zd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfischer-zd/subscriptions","organizations_url":"https://api.github.com/users/mfischer-zd/orgs","repos_url":"https://api.github.com/users/mfischer-zd/repos","events_url":"https://api.github.com/users/mfischer-zd/events{/privacy}","received_events_url":"https://api.github.com/users/mfischer-zd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-04T22:20:55Z","updated_at":"2015-08-05T15:57:44Z","closed_at":null,"body":"We saw this unusual chain of events reported by `consul lock` after sending it a SIGTERM today:\r\n\r\n```\r\n2015-08-03_22:39:06.20297 consul:Setting up lock at path: xxx\r\n2015-08-03_22:39:06.20308 consul:Attempting lock acquisition\r\n# Lock was never acquired because another candidate held it\r\n\r\n# SIGTERM was then sent\r\n2015-08-04_19:56:20.23976 consul:Shutdown triggered, killing child\r\n2015-08-04_19:56:20.23988 consul:No child process to kill\r\n# Correct, but then...\r\n2015-08-04_19:56:20.23992 consul:Starting handler 'myapp'\r\n# Wait, what?\r\n2015-08-04_19:56:20.25634 consul:Cleanup succeeded\r\n```\r\n\r\nAfter this, consul exited and  `myapp` was orphaned and reparented to PID 1."},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/128049071","html_url":"https://github.com/hashicorp/consul/issues/1155#issuecomment-128049071","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1155","id":128049071,"user":{"login":"mfischer-zd","id":984199,"avatar_url":"https://avatars.githubusercontent.com/u/984199?v=3","gravatar_id":"","url":"https://api.github.com/users/mfischer-zd","html_url":"https://github.com/mfischer-zd","followers_url":"https://api.github.com/users/mfischer-zd/followers","following_url":"https://api.github.com/users/mfischer-zd/following{/other_user}","gists_url":"https://api.github.com/users/mfischer-zd/gists{/gist_id}","starred_url":"https://api.github.com/users/mfischer-zd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfischer-zd/subscriptions","organizations_url":"https://api.github.com/users/mfischer-zd/orgs","repos_url":"https://api.github.com/users/mfischer-zd/repos","events_url":"https://api.github.com/users/mfischer-zd/events{/privacy}","received_events_url":"https://api.github.com/users/mfischer-zd/received_events","type":"User","site_admin":false},"created_at":"2015-08-05T15:57:44Z","updated_at":"2015-08-05T15:57:44Z","body":"I've investigated this matter further and it appears to be a race condition that can occur when a SIGTERM is sent to multiple consul lock processes at approximately the same time.  \r\n\r\nThe short story is this:\r\n\r\n1. host1 (current lock holder) receives SIGTERM and releases the lock\r\n2. host2 acquires the lock and runs LockCommand.startChild() in a goroutine\r\n3. host2: in startChild(), cmd.Start() is reached, forks and execs app\r\n4. host2 receives SIGTERM\r\n5. host2 calls LockCommand.killChild()\r\n6. host2: in killChild() tries to evaluate cmd.Process, but it hasn't been set yet in the goroutine running startChild()\r\n7. host2 exits, leaving an orphaned app"}}