{"action":"closed","number":11873,"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873","id":40867278,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11873","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11873.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11873.patch","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11873","number":11873,"state":"closed","locked":false,"title":"Fix flaky PD test by adding wait for detach","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"body":"Fixes #11302 \r\n\r\nFixes flakiness of `Pod Disks should schedule a pod w/ a readonly PD on two hosts, then remove both.` E2E test.\r\n\r\nThe flake boils down to a race condition in the test: The tests creates a PD and attaches it to node 0 in RW mode. Once this happens the PD can not be attached to any other GCE instance (a PD can only be attached to multiple instances if it is attached in read-only mode). The test deletes the RW PD pod and then immediately scheduled a pod on node 0 and node 1 that use the PD in read-only mode. Most of the time, the PD is detached from node 0 before the RO pod is scheduled to node 0, and all is well (node 0 detaches the PD and both node 0 and node 1 can then successfully attach the PD in RO mode). However, every now and then the test schedules the RO pod on node 0 right before the PD is declared an orphan and detached, and since the PD is the same, the RO pod is happily scheduled on node 0 (and until it is deleted, which won’t happen until both RO pods on both nodes are created, the PD remains attached to node 0 in RW mode). And since the PD is still attached in RW mode, it will never successfully attach to node 1, hence the test timeouts waiting for the second RO pod.\r\n\r\nThe fix is simply for the test to make sure that the disk is actually detached after the RW pod is deleted, before scheduling the RO pods.\r\n\r\nRunning test overnight to verify fix.\r\n\r\nDistilled repro log:\r\n```\r\n00:31:12.7251231126 test-runner: RW Pod -- submitting create request to e2e-test-saadali-minion-lbfb.\r\n00:31:12.776782 minion-lbfb: Attempting to attach GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\" in RW mode\r\n00:31:15.395904 minion-lbfb: Successfully attached GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\" in RW mode\r\n00:31:17.7251231176 test-runner: RW Pod -- created successfully. Submitting delete request.\r\n00:31:17.7251231176 test-runner: RW Pod -- delete request submitted.\r\n00:31:17.7251231176 test-runner: host0ROPod -- submitting create request to e2e-test-saadali-minion-lbfb.\r\n00:31:17.7251231176 test-runner: host1ROPod -- submitting create request to e2e-test-saadali-minion-ylen.\r\n00:31:17.943890 minion-lbfb: Attempting to attach GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\" in RO mode\r\n00:31:18.515354 minion-lbfb: Successfully attached GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\" in RO mode\r\n00:31:23.7251231236 test-runner: host0ROPod -- created successfully.\r\n00:36:23.7251231236 test-runner: host1ROPod -- timed out waiting for pod to be created\r\n00:36:35.690377 minion-lbfb: Attempting to detach GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\" in RO mode\r\n00:36:43.356365 minion-lbfb: Successfully detached GCE PD \"e2e-714f4fd7-3264-11e5-b33f-f0921cde18c1\".\r\n```","created_at":"2015-07-27T03:11:35Z","updated_at":"2015-07-28T22:23:11Z","closed_at":"2015-07-28T22:23:11Z","merged_at":"2015-07-28T22:23:11Z","merge_commit_sha":"96ce50f24b27102cf8e78df087c66017cbe89cd1","assignee":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873/commits","review_comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873/comments","review_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11873/comments","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/09c137e99b2266322613dd112fa0fe8a5e91ff3a","head":{"label":"saad-ali:11302FlakyPDTest","ref":"11302FlakyPDTest","sha":"09c137e99b2266322613dd112fa0fe8a5e91ff3a","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"repo":{"id":27614790,"name":"kubernetes","full_name":"saad-ali/kubernetes","owner":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/saad-ali/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/saad-ali/kubernetes","forks_url":"https://api.github.com/repos/saad-ali/kubernetes/forks","keys_url":"https://api.github.com/repos/saad-ali/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/saad-ali/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/saad-ali/kubernetes/teams","hooks_url":"https://api.github.com/repos/saad-ali/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/saad-ali/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/saad-ali/kubernetes/events","assignees_url":"https://api.github.com/repos/saad-ali/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/saad-ali/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/saad-ali/kubernetes/tags","blobs_url":"https://api.github.com/repos/saad-ali/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/saad-ali/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/saad-ali/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/saad-ali/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/saad-ali/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/saad-ali/kubernetes/languages","stargazers_url":"https://api.github.com/repos/saad-ali/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/saad-ali/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/saad-ali/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/saad-ali/kubernetes/subscription","commits_url":"https://api.github.com/repos/saad-ali/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/saad-ali/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/saad-ali/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/saad-ali/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/saad-ali/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/saad-ali/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/saad-ali/kubernetes/merges","archive_url":"https://api.github.com/repos/saad-ali/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/saad-ali/kubernetes/downloads","issues_url":"https://api.github.com/repos/saad-ali/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/saad-ali/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/saad-ali/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/saad-ali/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/saad-ali/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/saad-ali/kubernetes/releases{/id}","created_at":"2014-12-05T23:31:31Z","updated_at":"2015-05-07T05:23:04Z","pushed_at":"2015-07-28T18:26:04Z","git_url":"git://github.com/saad-ali/kubernetes.git","ssh_url":"git@github.com:saad-ali/kubernetes.git","clone_url":"https://github.com/saad-ali/kubernetes.git","svn_url":"https://github.com/saad-ali/kubernetes","homepage":null,"size":93650,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"GoogleCloudPlatform:master","ref":"master","sha":"c5bffaaf3166513da6259c44a5d1ba8e86bea5ce","user":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"GoogleCloudPlatform/kubernetes","owner":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes","forks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/forks","keys_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/teams","hooks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/events","assignees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/tags","blobs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/languages","stargazers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscription","commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/merges","archive_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/downloads","issues_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-07-28T21:18:49Z","pushed_at":"2015-07-28T22:23:11Z","git_url":"git://github.com/GoogleCloudPlatform/kubernetes.git","ssh_url":"git@github.com:GoogleCloudPlatform/kubernetes.git","clone_url":"https://github.com/GoogleCloudPlatform/kubernetes.git","svn_url":"https://github.com/GoogleCloudPlatform/kubernetes","homepage":"http://kubernetes.io","size":335191,"stargazers_count":9419,"watchers_count":9419,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2169,"mirror_url":null,"open_issues_count":1680,"forks":2169,"open_issues":1680,"watchers":9419,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11873"},"issue":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11873"},"comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11873/comments"},"review_comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873/comments"},"review_comment":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11873/commits"},"statuses":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/09c137e99b2266322613dd112fa0fe8a5e91ff3a"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"comments":13,"review_comments":0,"commits":1,"additions":36,"deletions":0,"changed_files":1}}