{"action":"created","comment":{"url":"https://api.github.com/repos/docker/distribution/pulls/comments/29071454","id":29071454,"diff_hunk":"@@ -394,18 +395,64 @@ func (d *driver) WriteStream(path string, offset int64, reader io.Reader) (total\n \t\tgo func(bytesRead int, from int64, buf []byte) {\n \t\t\tdefer d.putbuf(buf) // this buffer gets dropped after this call\n \n-\t\t\t// parts and partNumber are safe, because this function is the only one modifying them and we\n-\t\t\t// force it to be executed serially.\n-\t\t\tif bytesRead > 0 {\n-\t\t\t\tpart, putErr := multi.PutPart(int(partNumber), bytes.NewReader(buf[0:int64(bytesRead)+from]))\n-\t\t\t\tif putErr != nil {\n-\t\t\t\t\tputErrChan <- putErr\n+\t\t\t// DRAGONS(stevvooe): There are few things one might want to know\n+\t\t\t// about this section. First, the putErrChan is expecting an error\n+\t\t\t// and a nil or just a nil to come through the channel. This is\n+\t\t\t// covered by the silly defer below. The other aspect is the s3\n+\t\t\t// retry backoff to deal with RequestTimeout errors. Even though\n+\t\t\t// the underlying s3 library should handle it, it doesn't seem to\n+\t\t\t// be part of the shouldRetry function (see AdRoll/goamz/s3).\n+\t\t\tdefer func() {\n+\t\t\t\tputErrChan <- nil // for some reason, we do this no matter what.\n+\t\t\t}()\n+\n+\t\t\tif bytesRead <= 0 {\n+\t\t\t\treturn\n+\t\t\t}\n+\n+\t\t\tvar err error\n+\t\t\tvar part s3.Part\n+\n+\t\tloop:\n+\t\t\tfor retries := 0; retries < 5; retries++ {\n+\t\t\t\tpart, err = multi.PutPart(int(partNumber), bytes.NewReader(buf[0:int64(bytesRead)+from]))\n+\t\t\t\tif err == nil {\n+\t\t\t\t\tbreak // success!\n \t\t\t\t}\n \n-\t\t\t\tparts = append(parts, part)\n-\t\t\t\tpartNumber++\n+\t\t\t\t// NOTE(stevvooe): This retry code tries to only retry under\n+\t\t\t\t// conditions where the s3 package does not. We may add s3\n+\t\t\t\t// error codes to the below if we see others bubble up in the\n+\t\t\t\t// application. Right now, the most troubling is\n+\t\t\t\t// RequestTimeout, which seems to only triggered when a tcp\n+\t\t\t\t// connection to s3 slows to a crawl. If the RequestTimeout\n+\t\t\t\t// ends up getting added to the s3 library and we don't see\n+\t\t\t\t// other errors, this retry loop can be removed.\n+\t\t\t\tswitch err := err.(type) {\n+\t\t\t\tcase *s3.Error:\n+\t\t\t\t\tswitch err.Code {\n+\t\t\t\t\tcase \"RequestTimeout\":\n+\t\t\t\t\t\t// allow retries on only this error.\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tbreak loop\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tbackoff := 100 * time.Millisecond * time.Duration(retries+1)","path":"registry/storage/driver/s3/s3.go","position":63,"original_position":63,"commit_id":"e6715c5cec86aa7bc09390870e3f4f88eaba4b54","original_commit_id":"e6715c5cec86aa7bc09390870e3f4f88eaba4b54","user":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"body":"@BrianBland This is a good rule of thumb. Since these aren't necessarily \"bursty\" failures, exponential, randomized backoff isn't necessary here.\r\n\r\nWe'll take another pass at this when we clean it up.","created_at":"2015-04-24T18:07:15Z","updated_at":"2015-04-24T18:07:15Z","html_url":"https://github.com/docker/distribution/pull/430#discussion_r29071454","pull_request_url":"https://api.github.com/repos/docker/distribution/pulls/430","_links":{"self":{"href":"https://api.github.com/repos/docker/distribution/pulls/comments/29071454"},"html":{"href":"https://github.com/docker/distribution/pull/430#discussion_r29071454"},"pull_request":{"href":"https://api.github.com/repos/docker/distribution/pulls/430"}}},"pull_request":{"url":"https://api.github.com/repos/docker/distribution/pulls/430","id":34015512,"html_url":"https://github.com/docker/distribution/pull/430","diff_url":"https://github.com/docker/distribution/pull/430.diff","patch_url":"https://github.com/docker/distribution/pull/430.patch","issue_url":"https://api.github.com/repos/docker/distribution/issues/430","number":430,"state":"open","locked":false,"title":"Attempt to address intermittent s3 RequestTimeout error","user":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"body":"Addresses #413. Only close #413 after we have confirmed 400/500 errors have subsided on hub.\r\n\r\nSigned-off-by: Stephen J Day <stephen.day@docker.com>","created_at":"2015-04-24T03:08:39Z","updated_at":"2015-04-24T18:07:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bfbcb94175d1cd10c949a6de0558f96d256b2b55","assignee":null,"milestone":{"url":"https://api.github.com/repos/docker/distribution/milestones/11","html_url":"https://github.com/docker/distribution/milestones/Registry/2.0.1","labels_url":"https://api.github.com/repos/docker/distribution/milestones/11/labels","id":1071448,"number":11,"title":"Registry/2.0.1","description":"Issues and PRs marked on this milestone should be a part of the 2.0.1 release. No date will be set until we are certain this is necessary.","creator":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":17,"state":"open","created_at":"2015-04-16T22:43:18Z","updated_at":"2015-04-24T17:53:40Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/docker/distribution/pulls/430/commits","review_comments_url":"https://api.github.com/repos/docker/distribution/pulls/430/comments","review_comment_url":"https://api.github.com/repos/docker/distribution/pulls/comments{/number}","comments_url":"https://api.github.com/repos/docker/distribution/issues/430/comments","statuses_url":"https://api.github.com/repos/docker/distribution/statuses/e6715c5cec86aa7bc09390870e3f4f88eaba4b54","head":{"label":"stevvooe:address-s3-timeout-error","ref":"address-s3-timeout-error","sha":"e6715c5cec86aa7bc09390870e3f4f88eaba4b54","user":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"repo":{"id":28724012,"name":"distribution","full_name":"stevvooe/distribution","owner":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/stevvooe/distribution","description":"The Docker toolset to pack, ship, store, and deliver content","fork":true,"url":"https://api.github.com/repos/stevvooe/distribution","forks_url":"https://api.github.com/repos/stevvooe/distribution/forks","keys_url":"https://api.github.com/repos/stevvooe/distribution/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stevvooe/distribution/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stevvooe/distribution/teams","hooks_url":"https://api.github.com/repos/stevvooe/distribution/hooks","issue_events_url":"https://api.github.com/repos/stevvooe/distribution/issues/events{/number}","events_url":"https://api.github.com/repos/stevvooe/distribution/events","assignees_url":"https://api.github.com/repos/stevvooe/distribution/assignees{/user}","branches_url":"https://api.github.com/repos/stevvooe/distribution/branches{/branch}","tags_url":"https://api.github.com/repos/stevvooe/distribution/tags","blobs_url":"https://api.github.com/repos/stevvooe/distribution/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stevvooe/distribution/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stevvooe/distribution/git/refs{/sha}","trees_url":"https://api.github.com/repos/stevvooe/distribution/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stevvooe/distribution/statuses/{sha}","languages_url":"https://api.github.com/repos/stevvooe/distribution/languages","stargazers_url":"https://api.github.com/repos/stevvooe/distribution/stargazers","contributors_url":"https://api.github.com/repos/stevvooe/distribution/contributors","subscribers_url":"https://api.github.com/repos/stevvooe/distribution/subscribers","subscription_url":"https://api.github.com/repos/stevvooe/distribution/subscription","commits_url":"https://api.github.com/repos/stevvooe/distribution/commits{/sha}","git_commits_url":"https://api.github.com/repos/stevvooe/distribution/git/commits{/sha}","comments_url":"https://api.github.com/repos/stevvooe/distribution/comments{/number}","issue_comment_url":"https://api.github.com/repos/stevvooe/distribution/issues/comments{/number}","contents_url":"https://api.github.com/repos/stevvooe/distribution/contents/{+path}","compare_url":"https://api.github.com/repos/stevvooe/distribution/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stevvooe/distribution/merges","archive_url":"https://api.github.com/repos/stevvooe/distribution/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stevvooe/distribution/downloads","issues_url":"https://api.github.com/repos/stevvooe/distribution/issues{/number}","pulls_url":"https://api.github.com/repos/stevvooe/distribution/pulls{/number}","milestones_url":"https://api.github.com/repos/stevvooe/distribution/milestones{/number}","notifications_url":"https://api.github.com/repos/stevvooe/distribution/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stevvooe/distribution/labels{/name}","releases_url":"https://api.github.com/repos/stevvooe/distribution/releases{/id}","created_at":"2015-01-02T19:42:04Z","updated_at":"2015-04-23T22:45:49Z","pushed_at":"2015-04-24T17:53:42Z","git_url":"git://github.com/stevvooe/distribution.git","ssh_url":"git@github.com:stevvooe/distribution.git","clone_url":"https://github.com/stevvooe/distribution.git","svn_url":"https://github.com/stevvooe/distribution","homepage":null,"size":2882,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"docker:master","ref":"master","sha":"c20417ce68487924ad5df741ba7f08efa7a8dfa2","user":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"repo":{"id":28366058,"name":"distribution","full_name":"docker/distribution","owner":{"login":"docker","id":5429470,"avatar_url":"https://avatars.githubusercontent.com/u/5429470?v=3","gravatar_id":"","url":"https://api.github.com/users/docker","html_url":"https://github.com/docker","followers_url":"https://api.github.com/users/docker/followers","following_url":"https://api.github.com/users/docker/following{/other_user}","gists_url":"https://api.github.com/users/docker/gists{/gist_id}","starred_url":"https://api.github.com/users/docker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/docker/subscriptions","organizations_url":"https://api.github.com/users/docker/orgs","repos_url":"https://api.github.com/users/docker/repos","events_url":"https://api.github.com/users/docker/events{/privacy}","received_events_url":"https://api.github.com/users/docker/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/docker/distribution","description":"The Docker toolset to pack, ship, store, and deliver content","fork":false,"url":"https://api.github.com/repos/docker/distribution","forks_url":"https://api.github.com/repos/docker/distribution/forks","keys_url":"https://api.github.com/repos/docker/distribution/keys{/key_id}","collaborators_url":"https://api.github.com/repos/docker/distribution/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/docker/distribution/teams","hooks_url":"https://api.github.com/repos/docker/distribution/hooks","issue_events_url":"https://api.github.com/repos/docker/distribution/issues/events{/number}","events_url":"https://api.github.com/repos/docker/distribution/events","assignees_url":"https://api.github.com/repos/docker/distribution/assignees{/user}","branches_url":"https://api.github.com/repos/docker/distribution/branches{/branch}","tags_url":"https://api.github.com/repos/docker/distribution/tags","blobs_url":"https://api.github.com/repos/docker/distribution/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/docker/distribution/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/docker/distribution/git/refs{/sha}","trees_url":"https://api.github.com/repos/docker/distribution/git/trees{/sha}","statuses_url":"https://api.github.com/repos/docker/distribution/statuses/{sha}","languages_url":"https://api.github.com/repos/docker/distribution/languages","stargazers_url":"https://api.github.com/repos/docker/distribution/stargazers","contributors_url":"https://api.github.com/repos/docker/distribution/contributors","subscribers_url":"https://api.github.com/repos/docker/distribution/subscribers","subscription_url":"https://api.github.com/repos/docker/distribution/subscription","commits_url":"https://api.github.com/repos/docker/distribution/commits{/sha}","git_commits_url":"https://api.github.com/repos/docker/distribution/git/commits{/sha}","comments_url":"https://api.github.com/repos/docker/distribution/comments{/number}","issue_comment_url":"https://api.github.com/repos/docker/distribution/issues/comments{/number}","contents_url":"https://api.github.com/repos/docker/distribution/contents/{+path}","compare_url":"https://api.github.com/repos/docker/distribution/compare/{base}...{head}","merges_url":"https://api.github.com/repos/docker/distribution/merges","archive_url":"https://api.github.com/repos/docker/distribution/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/docker/distribution/downloads","issues_url":"https://api.github.com/repos/docker/distribution/issues{/number}","pulls_url":"https://api.github.com/repos/docker/distribution/pulls{/number}","milestones_url":"https://api.github.com/repos/docker/distribution/milestones{/number}","notifications_url":"https://api.github.com/repos/docker/distribution/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/docker/distribution/labels{/name}","releases_url":"https://api.github.com/repos/docker/distribution/releases{/id}","created_at":"2014-12-22T23:49:26Z","updated_at":"2015-04-24T17:53:40Z","pushed_at":"2015-04-24T17:53:40Z","git_url":"git://github.com/docker/distribution.git","ssh_url":"git@github.com:docker/distribution.git","clone_url":"https://github.com/docker/distribution.git","svn_url":"https://github.com/docker/distribution","homepage":null,"size":6973,"stargazers_count":357,"watchers_count":357,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":94,"mirror_url":null,"open_issues_count":113,"forks":94,"open_issues":113,"watchers":357,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/docker/distribution/pulls/430"},"html":{"href":"https://github.com/docker/distribution/pull/430"},"issue":{"href":"https://api.github.com/repos/docker/distribution/issues/430"},"comments":{"href":"https://api.github.com/repos/docker/distribution/issues/430/comments"},"review_comments":{"href":"https://api.github.com/repos/docker/distribution/pulls/430/comments"},"review_comment":{"href":"https://api.github.com/repos/docker/distribution/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/docker/distribution/pulls/430/commits"},"statuses":{"href":"https://api.github.com/repos/docker/distribution/statuses/e6715c5cec86aa7bc09390870e3f4f88eaba4b54"}}}}