{"action":"closed","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/1043","labels_url":"https://api.github.com/repos/docker/distribution/issues/1043/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/1043/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/1043/events","html_url":"https://github.com/docker/distribution/issues/1043","id":108988984,"number":1043,"title":"Registry push new images fail; pull & push old images work","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-09-30T00:59:14Z","updated_at":"2015-10-06T22:23:24Z","closed_at":"2015-10-06T22:23:24Z","body":"Hello,\r\n\r\nI have a private registry set up and have recently added token authentication. Since then, I am able to pull images just fine. I can also push images if no changes have been made. However, for some reason I cannot push new images. I see the token authentication working correctly, but the docker daemon returns `unauthorized: access to the requested resource is not authorized`.\r\n\r\nThis works (pulling image):\r\n`docker pull docker.djsdomain.com/nginxlb`\r\nlogs:\r\n```\r\ntime=\"2015-09-30T00:34:23.342064871Z\" level=warning msg=\"error authorizing context: authorization token required\" environment=comm http.request.host=docker.djsdomain.com http.request.id=1ab7ae41-f091-4931-9f14-e721b10aca11 http.request.method=GET http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 service=registry version=v2.1.1 \r\n10.170.77.66 - - [30/Sep/2015:00:34:23 +0000] \"GET /v2/ HTTP/1.0\" 401 114 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\ntime=\"2015-09-30T00:34:24.404675555Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=400.545µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:24.904817573Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=463.742µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:25.228066514Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=383.052µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.376398271Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=382.799µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.378139972Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=330.828µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.398979059Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=260.41µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.540102392Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=311.222µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.872992716Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=409.681µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.885936308Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=286.13µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:26.900131507Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=234.084µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:27.053922649Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 redis.connect.duration=377.838µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:34:27.489012757Z\" level=info msg=\"response completed\" environment=comm http.request.host=docker.djsdomain.com http.request.id=cb95bf14-fd36-462c-bac3-588daa60ab8c http.request.method=GET http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/nginxlb/manifests/latest\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" http.response.contenttype=\"application/json; charset=utf-8\" http.response.duration=3.89709066s http.response.status=200 http.response.written=55457 instance.id=bcc70f6e-be34-4535-a6b8-33db5ca1e317 service=registry version=v2.1.1 \r\n10.170.77.66 - - [30/Sep/2015:00:34:23 +0000] \"GET /v2/nginxlb/manifests/latest HTTP/1.0\" 200 55457 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\n```\r\n\r\nThis doesnt (pushing new image):\r\n`docker pull docker.djsdomain.com/gleaner`\r\nlogs:\r\n```\r\ntime=\"2015-09-30T00:46:06.153478919Z\" level=warning msg=\"error authorizing context: authorization token required\" environment=comm http.request.host=docker.djsdomain.com http.request.id=1ab9dc70-57c5-4a83-a96b-b31e8cddfb26 http.request.method=GET http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" instance.id=fb3f5c85-1b1f-440f-a8cf-980693b3890b service=registry version=v2.1.1 \r\n10.170.1.33 - - [30/Sep/2015:00:46:06 +0000] \"GET /v2/ HTTP/1.0\" 401 114 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\ntime=\"2015-09-30T00:46:06.449904999Z\" level=info msg=\"redis: connect redis:6379\" environment=comm instance.id=fb3f5c85-1b1f-440f-a8cf-980693b3890b redis.connect.duration=237.219µs service=registry version=v2.1.1 \r\ntime=\"2015-09-30T00:46:06.789613017Z\" level=error msg=\"response completed with error\" auth.user.name=djenriquez environment=comm err.code=\"BLOB_UNKNOWN\" err.detail=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 err.message=\"blob unknown to registry\" http.request.host=docker.djsdomain.com http.request.id=6eaafbdb-723a-4fe9-9ffd-437c4b9a7596 http.request.method=HEAD http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/gleaner/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" http.response.contenttype=\"application/json; charset=utf-8\" http.response.duration=342.124672ms http.response.status=404 http.response.written=157 instance.id=fb3f5c85-1b1f-440f-a8cf-980693b3890b service=registry vars.digest=\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" vars.name=gleaner version=v2.1.1 \r\n10.170.77.66 - - [30/Sep/2015:00:46:06 +0000] \"HEAD /v2/gleaner/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.0\" 404 157 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\ntime=\"2015-09-30T00:46:07.633475737Z\" level=info msg=\"response completed\" environment=comm http.request.host=docker.djsdomain.com http.request.id=8ae72b2e-7c65-42c9-9744-1ab7960a6202 http.request.method=POST http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/gleaner/blobs/uploads/\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" http.response.duration=726.028114ms http.response.status=202 http.response.written=0 instance.id=fb3f5c85-1b1f-440f-a8cf-980693b3890b service=registry version=v2.1.1 \r\n10.170.77.66 - - [30/Sep/2015:00:46:06 +0000] \"POST /v2/gleaner/blobs/uploads/ HTTP/1.0\" 202 0 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\ntime=\"2015-09-30T00:46:07.673610623Z\" level=warning msg=\"error authorizing context: authorization token required\" environment=comm http.request.host=docker.djsdomain.com http.request.id=10736a0a-4476-4dfd-a916-05274a71fe3d http.request.method=PATCH http.request.remoteaddr=10.0.63.64 http.request.uri=\"/v2/gleaner/blobs/uploads/b292f28a-bb0c-4c3e-a302-763e8d227abd?_state=lQoJCURMDOHJ5B9hFH4z4NVvYhJQD9y3aoPnvCHrHQR7Ik5hbWUiOiJnbGVhbmVyIiwiVVVJRCI6ImIyOTJmMjhhLWJiMGMtNGMzZS1hMzAyLTc2M2U4ZDIyN2FiZCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNS0wOS0zMFQwMDo0NjowNi45MDk0MTY2NTRaIn0%3D\" http.request.useragent=\"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\" instance.id=fb3f5c85-1b1f-440f-a8cf-980693b3890b service=registry vars.name=gleaner vars.uuid=b292f28a-bb0c-4c3e-a302-763e8d227abd version=v2.1.1 \r\n10.170.77.66 - - [30/Sep/2015:00:46:07 +0000] \"PATCH /v2/gleaner/blobs/uploads/b292f28a-bb0c-4c3e-a302-763e8d227abd?_state=lQoJCURMDOHJ5B9hFH4z4NVvYhJQD9y3aoPnvCHrHQR7Ik5hbWUiOiJnbGVhbmVyIiwiVVVJRCI6ImIyOTJmMjhhLWJiMGMtNGMzZS1hMzAyLTc2M2U4ZDIyN2FiZCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNS0wOS0zMFQwMDo0NjowNi45MDk0MTY2NTRaIn0%3D HTTP/1.0\" 401 221 \"\" \"docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/4.0.9-boot2docker os/linux arch/amd64\"\r\n```\r\n\r\nreturns:\r\n```\r\nbash-3.2$ docker push docker.djsdomain.com/gleaner\r\nThe push refers to a repository [docker.djsdomain.com/gleaner] (len: 1)\r\nc2dce7213213: Pushing [==================================================>]     32 B/32 B\r\nunauthorized: access to the requested resource is not authorized\r\n```\r\n\r\nNow, oddly enough, if the image already exists in S3 and no changes are made, it goes through the `Image already exists` and completes. Its only when there are changes where it fails.\r\n\r\nHere are the logs from the authentication server, incase they're helpful:\r\n```\r\nI0930 00:47:27.582009       1 server.go:220] Auth request: {djenriquez:***@10.170.140.121:37572 {djenriquez pull,push repository gleaner}}\r\nI0930 00:47:27.586096       1 server.go:103] Authn static djenriquez -> true, %!s(<nil>)\r\nI0930 00:47:27.586165       1 acl.go:37] {djenriquez pull,push repository gleaner} matched {\"Match\":{\"account\":\"djenriquez\"},\"Actions\":[\"*\"]}\r\nI0930 00:47:27.586436       1 server.go:122] Authz static ACL {djenriquez pull,push repository gleaner} -> [pull push], %!s(<nil>)\r\nI0930 00:47:27.595800       1 server.go:184] New token for {djenriquez:***@10.170.140.121:37572 {djenriquez pull,push repository gleaner}}: {\"iss\":\"djsdomain ACME server\",\"sub\":\"djenriquez\",\"aud\":\"Docker Registry\",\"exp\":1443574947,\"nbf\":1443574046,\"iat\":1443574047,\"jti\":\"6833611524882594851\",\"access\":[{\"type\":\"repository\",\"name\":\"gleaner\",\"actions\":[\"pull\",\"push\"]}]}\r\nI0930 00:47:27.595879       1 server.go:250] {\"token\":\"<Removing token for security -DJ>\"}\r\n```\r\n\r\nAnd also my registry config:\r\n```\r\nversion: 0.1\r\nlog:\r\n  level: info\r\n  formatter: text\r\n  fields:\r\n    service: registry\r\n    environment: comm\r\nstorage:\r\n  cache:\r\n    blobdescriptor: redis\r\n  s3:\r\n    region: us-east-1\r\n    bucket: docker.djsdomain.com\r\nauth:\r\n  token:\r\n    realm: \"https://dockerauth.djsdomain.com/auth\"\r\n    service: \"Docker Registry\"\r\n    issuer: \"djsdomain ACME server\"\r\n    rootcertbundle: /etc/docker/ssl/star_djsdomain_com.pem\r\nhttp:\r\n  addr: :5000\r\n  secret: <my secret>\r\nredis:\r\n  addr: redis:6379\r\n```\r\n\r\nAny ideas why I can't push new images??"}}