{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/928","labels_url":"https://api.github.com/repos/docker/distribution/issues/928/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/928/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/928/events","html_url":"https://github.com/docker/distribution/issues/928","id":103298083,"number":928,"title":"Push to private registry fails with BLOB_UNKNOWN v2.1.1","user":{"login":"caladbolg7","id":6886253,"avatar_url":"https://avatars.githubusercontent.com/u/6886253?v=3","gravatar_id":"","url":"https://api.github.com/users/caladbolg7","html_url":"https://github.com/caladbolg7","followers_url":"https://api.github.com/users/caladbolg7/followers","following_url":"https://api.github.com/users/caladbolg7/following{/other_user}","gists_url":"https://api.github.com/users/caladbolg7/gists{/gist_id}","starred_url":"https://api.github.com/users/caladbolg7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caladbolg7/subscriptions","organizations_url":"https://api.github.com/users/caladbolg7/orgs","repos_url":"https://api.github.com/users/caladbolg7/repos","events_url":"https://api.github.com/users/caladbolg7/events{/privacy}","received_events_url":"https://api.github.com/users/caladbolg7/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-08-26T15:32:24Z","updated_at":"2015-08-28T00:08:53Z","closed_at":null,"body":"I have a private registry sitting behind a pair of nginx proxies. The first terminates the TLS, while the second simply routes any information between the registry (:443) and a separate docker container (:80). The config files for both, version info, and other relevant information is contained in the gist.\r\n\r\nhttps://gist.github.com/caladbolg7/56488f65c108e5ae7ca1\r\n\r\nI can curl to /v2/ and /v2/_catalog just fine, but any attempt at pushing to the registry ends in a single push then a 'blob upload unknown'. Personally, I'm not sure what exactly is happening, as I had this set up working initially with 2.0 around two weeks ago, but after retesting yesterday, I receive this error, regardless of version (2.0/2.0.1/2.1.0/2.1.1).\r\n\r\nCurrently, the private registry stores to the local file system.\r\n\r\nError log:\r\n```\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"78a5be0a-5d9a-4d14-b4cb-36d56efda82b\",\"http.request.method\":\"GET\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"authorizing request\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.08135772Z\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"78a5be0a-5d9a-4d14-b4cb-36d56efda82b\",\"http.request.method\":\"GET\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"http.response.contenttype\":\"application/json; charset=utf-8\",\"http.response.duration\":\"4.747105ms\",\"http.response.status\":200,\"http.response.written\":2,\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"info\",\"msg\":\"response completed\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.081767837Z\",\"version\":\"v2.1.1\"}\r\n172.17.0.47 - - [26/Aug/2015:14:43:02 +0000] \"GET /v2/ HTTP/1.0\" 200 2 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\"\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"f6ac44a9-d53e-40c2-ad27-3c590c964a7a\",\"http.request.method\":\"HEAD\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"authorizing request\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.54730768Z\",\"vars.digest\":\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"f6ac44a9-d53e-40c2-ad27-3c590c964a7a\",\"http.request.method\":\"HEAD\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"GetBlob\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.547555181Z\",\"vars.digest\":\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"f6ac44a9-d53e-40c2-ad27-3c590c964a7a\",\"http.request.method\":\"HEAD\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"filesystem.GetContent(\\\"/docker/registry/v2/repositories/ubuntu2/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link\\\")\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.547881629Z\",\"trace.duration\":89812,\"trace.file\":\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\",\"trace.func\":\"github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent\",\"trace.id\":\"c8ef96a7-d0e9-445c-9c01-cc32c10c3da4\",\"trace.line\":55,\"vars.digest\":\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"err.code\":\"BLOB_UNKNOWN\",\"err.detail\":\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"err.message\":\"blob unknown to registry\",\"http.request.host\":\"--------------\",\"http.request.id\":\"f6ac44a9-d53e-40c2-ad27-3c590c964a7a\",\"http.request.method\":\"HEAD\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"http.response.contenttype\":\"application/json; charset=utf-8\",\"http.response.duration\":\"4.199659ms\",\"http.response.status\":404,\"http.response.written\":157,\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"error\",\"msg\":\"response completed with error\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.548199108Z\",\"vars.digest\":\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n172.17.0.47 - - [26/Aug/2015:14:43:02 +0000] \"HEAD /v2/ubuntu2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.0\" 404 157 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\"\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"e041587f-c7c8-403a-a193-cbd0ae4b27a8\",\"http.request.method\":\"POST\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/uploads/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"authorizing request\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.948769616Z\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"e041587f-c7c8-403a-a193-cbd0ae4b27a8\",\"http.request.method\":\"POST\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/uploads/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"(*linkedBlobStore).Writer\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.949089633Z\",\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"e041587f-c7c8-403a-a193-cbd0ae4b27a8\",\"http.request.method\":\"POST\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/uploads/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"filesystem.PutContent(\\\"/docker/registry/v2/repositories/ubuntu2/_uploads/072e3e64-1d82-421c-b7cd-c6f9be4c3fa4/startedat\\\")\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.950257626Z\",\"trace.duration\":914106,\"trace.file\":\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\",\"trace.func\":\"github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent\",\"trace.id\":\"8a59b089-6e54-4818-bd09-d13265c467fb\",\"trace.line\":67,\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"e041587f-c7c8-403a-a193-cbd0ae4b27a8\",\"http.request.method\":\"POST\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/uploads/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"debug\",\"msg\":\"filesystem.Stat(\\\"/docker/registry/v2/repositories/ubuntu2/_uploads/072e3e64-1d82-421c-b7cd-c6f9be4c3fa4/data\\\")\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.950585909Z\",\"trace.duration\":50927,\"trace.file\":\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\",\"trace.func\":\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat\",\"trace.id\":\"a60c9f24-4afc-4342-a91a-00a5c8df567a\",\"trace.line\":111,\"vars.name\":\"ubuntu2\",\"version\":\"v2.1.1\"}\r\n{\"http.request.host\":\"--------------\",\"http.request.id\":\"e041587f-c7c8-403a-a193-cbd0ae4b27a8\",\"http.request.method\":\"POST\",\"http.request.remoteaddr\":\"192.168.0.68\",\"http.request.uri\":\"/v2/ubuntu2/blobs/uploads/\",\"http.request.useragent\":\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\",\"http.response.duration\":\"11.405644ms\",\"http.response.status\":202,\"http.response.written\":0,\"instance.id\":\"4354d02c-5be5-45d4-a189-c1832b51daef\",\"level\":\"info\",\"msg\":\"response completed\",\"service\":\"registry\",\"time\":\"2015-08-26T14:43:02.954582117Z\",\"version\":\"v2.1.1\"}\r\n172.17.0.47 - - [26/Aug/2015:14:43:02 +0000] \"POST /v2/ubuntu2/blobs/uploads/ HTTP/1.0\" 202 0 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64\"\r\n```\r\n\r\nI'm sure it's likely something I'm glossing over, but if there's any extra info I need to supply I'll get it as quickly as possible.\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/135588890","html_url":"https://github.com/docker/distribution/issues/928#issuecomment-135588890","issue_url":"https://api.github.com/repos/docker/distribution/issues/928","id":135588890,"user":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"created_at":"2015-08-28T00:08:53Z","updated_at":"2015-08-28T00:08:53Z","body":"@caladbolg7 I don't think the push is actually succeeding at all, based on the registry logs. I think the upload request is getting corrupted and getting that 404. We'll get the patch in and give this a try."}}