{"action":"created","comment":{"url":"https://api.github.com/repos/square/p2/pulls/comments/47713755","id":47713755,"diff_hunk":"@@ -238,10 +230,16 @@ func (u update) lockRCs() error {\n \t}\n \n \terr = u.lock.Lock(oldPath)\n+\tif err != nil {\n+\t\t// The second key couldn't be locked, so release the first key before retrying.\n+\t\tRetryOrQuit(\n+\t\t\tfunc() error { return u.lock.Unlock(newPath) },\n+\t\t\tdone,\n+\t\t\tu.logger,\n+\t\t\tfmt.Sprintf(\"unlocking %s\", newPath),\n+\t\t)\n+\t}\n \tif _, ok := err.(kp.AlreadyLockedError); ok {","path":"pkg/roll/run_update.go","position":62,"original_position":62,"commit_id":"5faadb52b6028fa7b68784688d2ef24f6a783e69","original_commit_id":"5faadb52b6028fa7b68784688d2ef24f6a783e69","user":{"login":"bwester","id":1501338,"avatar_url":"https://avatars.githubusercontent.com/u/1501338?v=3","gravatar_id":"","url":"https://api.github.com/users/bwester","html_url":"https://github.com/bwester","followers_url":"https://api.github.com/users/bwester/followers","following_url":"https://api.github.com/users/bwester/following{/other_user}","gists_url":"https://api.github.com/users/bwester/gists{/gist_id}","starred_url":"https://api.github.com/users/bwester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwester/subscriptions","organizations_url":"https://api.github.com/users/bwester/orgs","repos_url":"https://api.github.com/users/bwester/repos","events_url":"https://api.github.com/users/bwester/events{/privacy}","received_events_url":"https://api.github.com/users/bwester/received_events","type":"User","site_admin":false},"body":"If no lock was acquired in the first place, control flow shouldn't come here :)\r\n\r\nBut if you're asking, \"Does `Unlock()` return an error if the lock isn't held?\" the answer is maybe.  If the key is not held by any session, an attempt to unlock it will succeed.  If the lock is held by a different session, it's an error.  If control flow makes it here and the lock is held by a different sessions, the locker's session must have been lost.","created_at":"2015-12-15T23:09:44Z","updated_at":"2015-12-15T23:09:44Z","html_url":"https://github.com/square/p2/pull/347#discussion_r47713755","pull_request_url":"https://api.github.com/repos/square/p2/pulls/347","_links":{"self":{"href":"https://api.github.com/repos/square/p2/pulls/comments/47713755"},"html":{"href":"https://github.com/square/p2/pull/347#discussion_r47713755"},"pull_request":{"href":"https://api.github.com/repos/square/p2/pulls/347"}}},"pull_request":{"url":"https://api.github.com/repos/square/p2/pulls/347","id":53642635,"html_url":"https://github.com/square/p2/pull/347","diff_url":"https://github.com/square/p2/pull/347.diff","patch_url":"https://github.com/square/p2/pull/347.patch","issue_url":"https://api.github.com/repos/square/p2/issues/347","number":347,"state":"open","locked":false,"title":"Always release RC locks in a rolling update","user":{"login":"bwester","id":1501338,"avatar_url":"https://avatars.githubusercontent.com/u/1501338?v=3","gravatar_id":"","url":"https://api.github.com/users/bwester","html_url":"https://github.com/bwester","followers_url":"https://api.github.com/users/bwester/followers","following_url":"https://api.github.com/users/bwester/following{/other_user}","gists_url":"https://api.github.com/users/bwester/gists{/gist_id}","starred_url":"https://api.github.com/users/bwester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwester/subscriptions","organizations_url":"https://api.github.com/users/bwester/orgs","repos_url":"https://api.github.com/users/bwester/repos","events_url":"https://api.github.com/users/bwester/events{/privacy}","received_events_url":"https://api.github.com/users/bwester/received_events","type":"User","site_admin":false},"body":"When a rolling update executes, it acquires locks on the RCs involved in the\r\nchange, and those locks should be released when the update is finished.\r\nHowever, the existing code never verifies that the locks are actually released.\r\nTransient errors could allow the updater to leak a locked RC, preventing\r\nanything from modifying that RC until the updater's Consul session is\r\nrestarted.\r\n\r\nThis change reduces the capacity for deadlocks among rolling updates by\r\nretrying all unlock operations. As long as Consul is accessible, the unlocks\r\nshould eventually succeed. During prolonged Consul outages, the updater's\r\nsession could expire, also dropping the locks and allowing the rolling update\r\nto restart.","created_at":"2015-12-15T00:15:23Z","updated_at":"2015-12-15T23:09:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"415dec6898957c658bbce66b2670e60b14ec1ef0","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/square/p2/pulls/347/commits","review_comments_url":"https://api.github.com/repos/square/p2/pulls/347/comments","review_comment_url":"https://api.github.com/repos/square/p2/pulls/comments{/number}","comments_url":"https://api.github.com/repos/square/p2/issues/347/comments","statuses_url":"https://api.github.com/repos/square/p2/statuses/5faadb52b6028fa7b68784688d2ef24f6a783e69","head":{"label":"bwester:lock-release","ref":"lock-release","sha":"5faadb52b6028fa7b68784688d2ef24f6a783e69","user":{"login":"bwester","id":1501338,"avatar_url":"https://avatars.githubusercontent.com/u/1501338?v=3","gravatar_id":"","url":"https://api.github.com/users/bwester","html_url":"https://github.com/bwester","followers_url":"https://api.github.com/users/bwester/followers","following_url":"https://api.github.com/users/bwester/following{/other_user}","gists_url":"https://api.github.com/users/bwester/gists{/gist_id}","starred_url":"https://api.github.com/users/bwester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwester/subscriptions","organizations_url":"https://api.github.com/users/bwester/orgs","repos_url":"https://api.github.com/users/bwester/repos","events_url":"https://api.github.com/users/bwester/events{/privacy}","received_events_url":"https://api.github.com/users/bwester/received_events","type":"User","site_admin":false},"repo":{"id":33944046,"name":"p2","full_name":"bwester/p2","owner":{"login":"bwester","id":1501338,"avatar_url":"https://avatars.githubusercontent.com/u/1501338?v=3","gravatar_id":"","url":"https://api.github.com/users/bwester","html_url":"https://github.com/bwester","followers_url":"https://api.github.com/users/bwester/followers","following_url":"https://api.github.com/users/bwester/following{/other_user}","gists_url":"https://api.github.com/users/bwester/gists{/gist_id}","starred_url":"https://api.github.com/users/bwester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwester/subscriptions","organizations_url":"https://api.github.com/users/bwester/orgs","repos_url":"https://api.github.com/users/bwester/repos","events_url":"https://api.github.com/users/bwester/events{/privacy}","received_events_url":"https://api.github.com/users/bwester/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bwester/p2","description":"Platypus Platform: Tools for Scalable Deployment","fork":true,"url":"https://api.github.com/repos/bwester/p2","forks_url":"https://api.github.com/repos/bwester/p2/forks","keys_url":"https://api.github.com/repos/bwester/p2/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bwester/p2/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bwester/p2/teams","hooks_url":"https://api.github.com/repos/bwester/p2/hooks","issue_events_url":"https://api.github.com/repos/bwester/p2/issues/events{/number}","events_url":"https://api.github.com/repos/bwester/p2/events","assignees_url":"https://api.github.com/repos/bwester/p2/assignees{/user}","branches_url":"https://api.github.com/repos/bwester/p2/branches{/branch}","tags_url":"https://api.github.com/repos/bwester/p2/tags","blobs_url":"https://api.github.com/repos/bwester/p2/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bwester/p2/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bwester/p2/git/refs{/sha}","trees_url":"https://api.github.com/repos/bwester/p2/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bwester/p2/statuses/{sha}","languages_url":"https://api.github.com/repos/bwester/p2/languages","stargazers_url":"https://api.github.com/repos/bwester/p2/stargazers","contributors_url":"https://api.github.com/repos/bwester/p2/contributors","subscribers_url":"https://api.github.com/repos/bwester/p2/subscribers","subscription_url":"https://api.github.com/repos/bwester/p2/subscription","commits_url":"https://api.github.com/repos/bwester/p2/commits{/sha}","git_commits_url":"https://api.github.com/repos/bwester/p2/git/commits{/sha}","comments_url":"https://api.github.com/repos/bwester/p2/comments{/number}","issue_comment_url":"https://api.github.com/repos/bwester/p2/issues/comments{/number}","contents_url":"https://api.github.com/repos/bwester/p2/contents/{+path}","compare_url":"https://api.github.com/repos/bwester/p2/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bwester/p2/merges","archive_url":"https://api.github.com/repos/bwester/p2/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bwester/p2/downloads","issues_url":"https://api.github.com/repos/bwester/p2/issues{/number}","pulls_url":"https://api.github.com/repos/bwester/p2/pulls{/number}","milestones_url":"https://api.github.com/repos/bwester/p2/milestones{/number}","notifications_url":"https://api.github.com/repos/bwester/p2/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bwester/p2/labels{/name}","releases_url":"https://api.github.com/repos/bwester/p2/releases{/id}","created_at":"2015-04-14T16:40:53Z","updated_at":"2015-05-04T22:31:34Z","pushed_at":"2015-12-15T22:37:37Z","git_url":"git://github.com/bwester/p2.git","ssh_url":"git@github.com:bwester/p2.git","clone_url":"https://github.com/bwester/p2.git","svn_url":"https://github.com/bwester/p2","homepage":"","size":31003,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"square:master","ref":"master","sha":"d382f5e6d8418223d6767142fd69558a4ae81c02","user":{"login":"square","id":82592,"avatar_url":"https://avatars.githubusercontent.com/u/82592?v=3","gravatar_id":"","url":"https://api.github.com/users/square","html_url":"https://github.com/square","followers_url":"https://api.github.com/users/square/followers","following_url":"https://api.github.com/users/square/following{/other_user}","gists_url":"https://api.github.com/users/square/gists{/gist_id}","starred_url":"https://api.github.com/users/square/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/square/subscriptions","organizations_url":"https://api.github.com/users/square/orgs","repos_url":"https://api.github.com/users/square/repos","events_url":"https://api.github.com/users/square/events{/privacy}","received_events_url":"https://api.github.com/users/square/received_events","type":"Organization","site_admin":false},"repo":{"id":25320424,"name":"p2","full_name":"square/p2","owner":{"login":"square","id":82592,"avatar_url":"https://avatars.githubusercontent.com/u/82592?v=3","gravatar_id":"","url":"https://api.github.com/users/square","html_url":"https://github.com/square","followers_url":"https://api.github.com/users/square/followers","following_url":"https://api.github.com/users/square/following{/other_user}","gists_url":"https://api.github.com/users/square/gists{/gist_id}","starred_url":"https://api.github.com/users/square/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/square/subscriptions","organizations_url":"https://api.github.com/users/square/orgs","repos_url":"https://api.github.com/users/square/repos","events_url":"https://api.github.com/users/square/events{/privacy}","received_events_url":"https://api.github.com/users/square/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/square/p2","description":"Platypus Platform: Tools for Scalable Deployment","fork":false,"url":"https://api.github.com/repos/square/p2","forks_url":"https://api.github.com/repos/square/p2/forks","keys_url":"https://api.github.com/repos/square/p2/keys{/key_id}","collaborators_url":"https://api.github.com/repos/square/p2/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/square/p2/teams","hooks_url":"https://api.github.com/repos/square/p2/hooks","issue_events_url":"https://api.github.com/repos/square/p2/issues/events{/number}","events_url":"https://api.github.com/repos/square/p2/events","assignees_url":"https://api.github.com/repos/square/p2/assignees{/user}","branches_url":"https://api.github.com/repos/square/p2/branches{/branch}","tags_url":"https://api.github.com/repos/square/p2/tags","blobs_url":"https://api.github.com/repos/square/p2/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/square/p2/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/square/p2/git/refs{/sha}","trees_url":"https://api.github.com/repos/square/p2/git/trees{/sha}","statuses_url":"https://api.github.com/repos/square/p2/statuses/{sha}","languages_url":"https://api.github.com/repos/square/p2/languages","stargazers_url":"https://api.github.com/repos/square/p2/stargazers","contributors_url":"https://api.github.com/repos/square/p2/contributors","subscribers_url":"https://api.github.com/repos/square/p2/subscribers","subscription_url":"https://api.github.com/repos/square/p2/subscription","commits_url":"https://api.github.com/repos/square/p2/commits{/sha}","git_commits_url":"https://api.github.com/repos/square/p2/git/commits{/sha}","comments_url":"https://api.github.com/repos/square/p2/comments{/number}","issue_comment_url":"https://api.github.com/repos/square/p2/issues/comments{/number}","contents_url":"https://api.github.com/repos/square/p2/contents/{+path}","compare_url":"https://api.github.com/repos/square/p2/compare/{base}...{head}","merges_url":"https://api.github.com/repos/square/p2/merges","archive_url":"https://api.github.com/repos/square/p2/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/square/p2/downloads","issues_url":"https://api.github.com/repos/square/p2/issues{/number}","pulls_url":"https://api.github.com/repos/square/p2/pulls{/number}","milestones_url":"https://api.github.com/repos/square/p2/milestones{/number}","notifications_url":"https://api.github.com/repos/square/p2/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/square/p2/labels{/name}","releases_url":"https://api.github.com/repos/square/p2/releases{/id}","created_at":"2014-10-16T20:31:12Z","updated_at":"2015-12-10T18:16:39Z","pushed_at":"2015-12-15T22:39:07Z","git_url":"git://github.com/square/p2.git","ssh_url":"git@github.com:square/p2.git","clone_url":"https://github.com/square/p2.git","svn_url":"https://github.com/square/p2","homepage":"","size":31206,"stargazers_count":88,"watchers_count":88,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":15,"mirror_url":null,"open_issues_count":7,"forks":15,"open_issues":7,"watchers":88,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/square/p2/pulls/347"},"html":{"href":"https://github.com/square/p2/pull/347"},"issue":{"href":"https://api.github.com/repos/square/p2/issues/347"},"comments":{"href":"https://api.github.com/repos/square/p2/issues/347/comments"},"review_comments":{"href":"https://api.github.com/repos/square/p2/pulls/347/comments"},"review_comment":{"href":"https://api.github.com/repos/square/p2/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/square/p2/pulls/347/commits"},"statuses":{"href":"https://api.github.com/repos/square/p2/statuses/5faadb52b6028fa7b68784688d2ef24f6a783e69"}}}}