{"action":"created","comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/35498681","id":35498681,"diff_hunk":"@@ -1710,6 +1733,31 @@ func (s *AWSCloud) ensureSecurityGroup(name string, description string, vpcID st\n \treturn groupID, nil\n }\n \n+// createTags calls EC2 CreateTags, but adds retry-on-failure logic\n+// We retry mainly because if we create an object, we cannot tag it until it is \"fully created\" (eventual consistency)\n+// The error code varies though (depending on what we are tagging), so we simply retry on all errors\n+func (s *AWSCloud) createTags(request *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error) {\n+\tattempt := 0","path":"pkg/cloudprovider/aws/aws.go","position":110,"original_position":110,"commit_id":"54fd6acf9c42f27a981ff2ea82821a7615db9d0f","original_commit_id":"54fd6acf9c42f27a981ff2ea82821a7615db9d0f","user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"body":"I really should be doing exponential backoff here I think.  AWS had a pretty bad outage in the early days because everyone was just doing straight loops, and so a degraded API grew into a full on outage, as I recall.  I hope we can implement that using a reusable Go function but I'm not entirely sure how best to express that.  Maybe something like time.Tick but with a timeout and backoff?   I'll add a TODO to add exponential backoff in the meantime.","created_at":"2015-07-26T22:22:35Z","updated_at":"2015-07-26T22:22:35Z","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029#discussion_r35498681","pull_request_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029","_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/35498681"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029#discussion_r35498681"},"pull_request":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029"}}},"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029","id":39640566,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029.patch","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11029","number":11029,"state":"open","locked":false,"title":"AWS: Fix security group races","user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"body":"Fix two unlikely but hit-in-the-real-world (during conformance tests) race conditions in AWS, around eventual consistency in the AWS EC2 API for security groups.  Essentially retry a sensible number of times in the hope we eventually get a consistent view.","created_at":"2015-07-09T23:52:42Z","updated_at":"2015-07-26T22:22:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8fc7dd310ad630207e2c2321af3df8e24306a110","assignee":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029/commits","review_comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029/comments","review_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11029/comments","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/54fd6acf9c42f27a981ff2ea82821a7615db9d0f","head":{"label":"justinsb:fix_aws_security_group_races","ref":"fix_aws_security_group_races","sha":"54fd6acf9c42f27a981ff2ea82821a7615db9d0f","user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"repo":{"id":26103010,"name":"kubernetes","full_name":"justinsb/kubernetes","owner":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/justinsb/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/justinsb/kubernetes","forks_url":"https://api.github.com/repos/justinsb/kubernetes/forks","keys_url":"https://api.github.com/repos/justinsb/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/justinsb/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/justinsb/kubernetes/teams","hooks_url":"https://api.github.com/repos/justinsb/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/justinsb/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/justinsb/kubernetes/events","assignees_url":"https://api.github.com/repos/justinsb/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/justinsb/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/justinsb/kubernetes/tags","blobs_url":"https://api.github.com/repos/justinsb/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/justinsb/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/justinsb/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/justinsb/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/justinsb/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/justinsb/kubernetes/languages","stargazers_url":"https://api.github.com/repos/justinsb/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/justinsb/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/justinsb/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/justinsb/kubernetes/subscription","commits_url":"https://api.github.com/repos/justinsb/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/justinsb/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/justinsb/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/justinsb/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/justinsb/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/justinsb/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/justinsb/kubernetes/merges","archive_url":"https://api.github.com/repos/justinsb/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/justinsb/kubernetes/downloads","issues_url":"https://api.github.com/repos/justinsb/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/justinsb/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/justinsb/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/justinsb/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/justinsb/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/justinsb/kubernetes/releases{/id}","created_at":"2014-11-03T03:53:05Z","updated_at":"2015-02-24T20:55:05Z","pushed_at":"2015-07-26T21:16:08Z","git_url":"git://github.com/justinsb/kubernetes.git","ssh_url":"git@github.com:justinsb/kubernetes.git","clone_url":"https://github.com/justinsb/kubernetes.git","svn_url":"https://github.com/justinsb/kubernetes","homepage":null,"size":92507,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"GoogleCloudPlatform:master","ref":"master","sha":"d864d271af6858738da13cbdb76791050257217a","user":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"GoogleCloudPlatform/kubernetes","owner":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes","forks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/forks","keys_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/teams","hooks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/events","assignees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/tags","blobs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/languages","stargazers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscription","commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/merges","archive_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/downloads","issues_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-07-26T21:56:15Z","pushed_at":"2015-07-26T21:13:12Z","git_url":"git://github.com/GoogleCloudPlatform/kubernetes.git","ssh_url":"git@github.com:GoogleCloudPlatform/kubernetes.git","clone_url":"https://github.com/GoogleCloudPlatform/kubernetes.git","svn_url":"https://github.com/GoogleCloudPlatform/kubernetes","homepage":"http://kubernetes.io","size":332594,"stargazers_count":9353,"watchers_count":9353,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2144,"mirror_url":null,"open_issues_count":1665,"forks":2144,"open_issues":1665,"watchers":9353,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11029"},"issue":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11029"},"comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11029/comments"},"review_comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029/comments"},"review_comment":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11029/commits"},"statuses":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/54fd6acf9c42f27a981ff2ea82821a7615db9d0f"}}}}