{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/769","labels_url":"https://api.github.com/repos/docker/distribution/issues/769/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/769/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/769/events","html_url":"https://github.com/docker/distribution/issues/769","id":97640814,"number":769,"title":"Private registry push failed: 500 error due to S3 \"SlowDown\" error","user":{"login":"baldzern4","id":7126382,"avatar_url":"https://avatars.githubusercontent.com/u/7126382?v=3","gravatar_id":"","url":"https://api.github.com/users/baldzern4","html_url":"https://github.com/baldzern4","followers_url":"https://api.github.com/users/baldzern4/followers","following_url":"https://api.github.com/users/baldzern4/following{/other_user}","gists_url":"https://api.github.com/users/baldzern4/gists{/gist_id}","starred_url":"https://api.github.com/users/baldzern4/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/baldzern4/subscriptions","organizations_url":"https://api.github.com/users/baldzern4/orgs","repos_url":"https://api.github.com/users/baldzern4/repos","events_url":"https://api.github.com/users/baldzern4/events{/privacy}","received_events_url":"https://api.github.com/users/baldzern4/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/distribution/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/docker/distribution/labels/theme/quality","name":"theme/quality","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"RichardScothern","id":922018,"avatar_url":"https://avatars.githubusercontent.com/u/922018?v=3","gravatar_id":"","url":"https://api.github.com/users/RichardScothern","html_url":"https://github.com/RichardScothern","followers_url":"https://api.github.com/users/RichardScothern/followers","following_url":"https://api.github.com/users/RichardScothern/following{/other_user}","gists_url":"https://api.github.com/users/RichardScothern/gists{/gist_id}","starred_url":"https://api.github.com/users/RichardScothern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RichardScothern/subscriptions","organizations_url":"https://api.github.com/users/RichardScothern/orgs","repos_url":"https://api.github.com/users/RichardScothern/repos","events_url":"https://api.github.com/users/RichardScothern/events{/privacy}","received_events_url":"https://api.github.com/users/RichardScothern/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/docker/distribution/milestones/9","html_url":"https://github.com/docker/distribution/milestones/Registry/2.2","labels_url":"https://api.github.com/repos/docker/distribution/milestones/9/labels","id":1026605,"number":9,"title":"Registry/2.2","description":"","creator":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"open_issues":32,"closed_issues":6,"state":"open","created_at":"2015-03-18T04:28:58Z","updated_at":"2015-08-12T06:27:06Z","due_on":"2015-08-26T07:00:00Z","closed_at":null},"comments":10,"created_at":"2015-07-28T08:02:01Z","updated_at":"2015-08-12T14:50:41Z","closed_at":null,"body":"Hi,\r\nwe are running the latest private registry 2.0.1 with AWS S3 as backend. \r\nWhile uploading images (~100-300 MB) we are getting from time to time http:503 \"SlowDown\" errors from S3.\r\n\r\nThis error can happen if hundreds of PUT request are done to one bucket (see [http://aws.amazon.com/articles/1904]())\r\n\r\nSo I think:\r\n \r\n1. S3 Storage engine doesn't react on this error with exponational backoff strategy as AWS discribes in [http://aws.amazon.com/articles/1904]().\r\n2. The error is not handled correctly, because no retry is done. Instead the error is forwared as http:500 to the client.\r\n\r\n**Docker Registry Information**  \r\nDocker version: docker/1.7.1 go/go1.4.2 git-commit/786b29d \r\nStorage backend: S3 \r\n\r\n**Debug output:**  \r\n\r\n```\r\ntime=\"2015-07-24T09:30:51.2516226Z\" level=info msg=\"s3.WriteStream(\\\"/docker/registry/v2/repositories/company/n4servicebase_n4relp/_uploads/5892e837-1860-46a2-94cc-528223bacceb/data\\\", 0)\" trace.duration=12.908167038s trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base\r\n/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).WriteStream\" trace.id=266b94bc-81c7-48ec-b2a8-f631fd33ba57 trace.line=95 \r\n\r\n\r\ntime=\"2015-07-24T09:30:51.251691872Z\" level=debug msg=\"(*layerWriter).Finish\" environment=development http.request.host=<REMOVED> http.request.id=543789a7-bc1c-470f-ac4f-4a189e615f5a http.request.method=PUT http.request.remoteaddr=10.0.2.158 http.request.uri=\"/v2/company/n4servicebase_\r\nn4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>\r\n%3D&digest=sha256%<DIGEST>\" http.request.useragent=\"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\" instance.id=727f874c-b023-40df-b4d9-a36048868d9f service=registry vars.name=\"company/n4serviceb\r\nase_n4relp\" vars.uuid=5892e837-1860-46a2-94cc-528223bacceb version=v2.0.1 \r\n\r\ntime=\"2015-07-24T09:30:51.295113566Z\" level=info msg=\"s3.Stat(\\\"/docker/registry/v2/blobs/sha256/8a/8abb6c6a3c0a8b7d9ec837ddd87fd346a2a5b43b0dafcce6d5fac80984e14aaa/data\\\")\" trace.duration=43.315054ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func\r\n=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat\" trace.id=1bd54ec7-64d2-4cdf-9de8-e42925262bcb trace.line=111 \r\n\r\ntime=\"2015-07-24T09:30:51.337778973Z\" level=info msg=\"s3.Stat(\\\"/docker/registry/v2/repositories/company/n4servicebase_n4relp/_uploads/5892e837-1860-46a2-94cc-528223bacceb/data\\\")\" trace.duration=42.560934ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" \r\ntrace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat\" trace.id=9639bf50-1cd3-487e-a593-cfc985648e30 trace.line=111 \r\n\r\ntime=\"2015-07-24T09:30:51.354908062Z\" level=info msg=\"s3.Move(\\\"/docker/registry/v2/repositories/company/n4servicebase_n4relp/_uploads/5892e837-1860-46a2-94cc-528223bacceb/data\\\", \\\"/docker/registry/v2/blobs/sha256/8a/8abb6c6a3c0a8b7d9ec837ddd87fd346a2a5b43b0dafcce6d5fac80984e14aaa/data\\\"\" trace.\r\nduration=17.031001ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Move\" trace.id=c2deb7f7-2b47-44c1-872d-cccf0f4593d8 trace.line=135 \r\n\r\ntime=\"2015-07-24T09:30:51.354986724Z\" level=error msg=\"unknown error completing upload: &s3.Error{StatusCode:503, Code:\\\"SlowDown\\\", Message:\\\"Please reduce your request rate.\\\", BucketName:\\\"\\\", RequestId:\\\"15498834E4967418\\\", HostId:\\\"88S6JEd1CNLl6w0O7kfoaKFAQzGcb/MRhZMQThNV4n4JGLKiTqYwCUSaDQ0hkwX/BL9SbmtSeXg=\\\"}\" environment=development http.request.host=<REMOVED> http.request.id=543789a7-bc1c-470f-ac4f-4a189e615f5a http.request.method=PUT http.request.remoteaddr=10.0.2.158 http.request.uri=\"/v2/company/n4servicebase_n4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>%3D&digest=sha256%<DIGEST>\" http.request.useragent=\"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\" instance.id=727f874c-b023-40df-b4d9-a36048868d9f service=registry vars.name=\"company/n4servicebase_n4relp\" vars.uuid=5892e837-1860-46a2-94cc-528223bacceb version=v2.0.1 \r\n\r\ntime=\"2015-07-24T09:30:51.355054525Z\" level=debug msg=\"(*layerWriter).Cancel\" environment=development http.request.host=<REMOVED> http.request.id=543789a7-bc1c-470f-ac4f-4a189e615f5a http.request.method=PUT http.request.remoteaddr=10.0.2.158 http.request.uri=\"/v2/company/n4servicebase_n4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>%3D&digest=sha256%<DIGEST>\" http.request.useragent=\"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\" instance.id=727f874c-b023-40df-b4d9-a36048868d9f service=registry vars.name=\"company/n4servicebase_n4relp\" vars.uuid=5892e837-1860-46a2-94cc-528223bacceb version=v2.0.1 \r\n\r\ntime=\"2015-07-24T09:30:51.49266334Z\" level=info msg=\"s3.Delete(\\\"/docker/registry/v2/repositories/company/n4servicebase_n4relp/_uploads/5892e837-1860-46a2-94cc-528223bacceb\\\")\" trace.duration=137.54242ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete\" trace.id=84ddcfeb-0afc-4ddf-a569-01597fd5fe80 trace.line=149 \r\n\r\ntime=\"2015-07-24T09:30:51.492750912Z\" level=error msg=\"An error occured\" environment=development err.code=UNKNOWN err.detail=\"Please reduce your request rate.\" err.message=\"unknown error\" http.request.host=<REMOVED> http.request.id=543789a7-bc1c-470f-ac4f-4a189e615f5a http.request.method=PUT http.request.remoteaddr=10.0.2.158 http.request.uri=\"/v2/company/n4servicebase_n4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>%3D&digest=sha256%<DIGEST>\" http.request.useragent=\"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\" instance.id=727f874c-b023-40df-b4d9-a36048868d9f service=registry vars.name=\"company/n4servicebase_n4relp\" vars.uuid=5892e837-1860-46a2-94cc-528223bacceb version=v2.0.1 \r\n\r\ntime=\"2015-07-24T09:30:51.49284106Z\" level=info msg=\"response completed\" environment=development http.request.host=<REMOVED> http.request.id=543789a7-bc1c-470f-ac4f-4a189e615f5a http.request.method=PUT http.request.remoteaddr=10.0.2.158 http.request.uri=\"/v2/company/n4servicebase_n4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>%3D&digest=sha256%<DIGEST>\" http.request.useragent=\"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\" http.response.contenttype=\"application/json; charset=utf-8\" http.response.duration=13.190556546s http.response.status=500 http.response.written=102 instance.id=727f874c-b023-40df-b4d9-a36048868d9f service=registry version=v2.0.1 \r\n\r\n172.31.8.163 - - [24/Jul/2015:09:30:38 +0000] \"PUT /v2/company/n4servicebase_n4relp/blobs/uploads/5892e837-1860-46a2-94cc-528223bacceb?_state=<STATE>%3D&digest=sha256%<DIGEST> HTTP/1.1\" 500 102 \"\" \"docker/1.7.1 go/go1.4.2 git-commit/786b29d kernel/3.16.0-0.bpo.4-amd64 os/linux arch/amd64\"\r\n```\r\n*Some content was replaced with tags to make the output readable. \r\n\r\n\r\nBr,  \r\nJörg Baldzer"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/130329256","html_url":"https://github.com/docker/distribution/issues/769#issuecomment-130329256","issue_url":"https://api.github.com/repos/docker/distribution/issues/769","id":130329256,"user":{"login":"baldzern4","id":7126382,"avatar_url":"https://avatars.githubusercontent.com/u/7126382?v=3","gravatar_id":"","url":"https://api.github.com/users/baldzern4","html_url":"https://github.com/baldzern4","followers_url":"https://api.github.com/users/baldzern4/followers","following_url":"https://api.github.com/users/baldzern4/following{/other_user}","gists_url":"https://api.github.com/users/baldzern4/gists{/gist_id}","starred_url":"https://api.github.com/users/baldzern4/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/baldzern4/subscriptions","organizations_url":"https://api.github.com/users/baldzern4/orgs","repos_url":"https://api.github.com/users/baldzern4/repos","events_url":"https://api.github.com/users/baldzern4/events{/privacy}","received_events_url":"https://api.github.com/users/baldzern4/received_events","type":"User","site_admin":false},"created_at":"2015-08-12T14:50:41Z","updated_at":"2015-08-12T14:50:41Z","body":"@RichardScothern I was able to push several images without any issues."}}