{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/785","labels_url":"https://api.github.com/repos/docker/distribution/issues/785/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/785/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/785/events","html_url":"https://github.com/docker/distribution/issues/785","id":98148068,"number":785,"title":"Intermittent \"connection reset by peer\" while pushing image","user":{"login":"pliljenberg","id":378483,"avatar_url":"https://avatars.githubusercontent.com/u/378483?v=3","gravatar_id":"","url":"https://api.github.com/users/pliljenberg","html_url":"https://github.com/pliljenberg","followers_url":"https://api.github.com/users/pliljenberg/followers","following_url":"https://api.github.com/users/pliljenberg/following{/other_user}","gists_url":"https://api.github.com/users/pliljenberg/gists{/gist_id}","starred_url":"https://api.github.com/users/pliljenberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pliljenberg/subscriptions","organizations_url":"https://api.github.com/users/pliljenberg/orgs","repos_url":"https://api.github.com/users/pliljenberg/repos","events_url":"https://api.github.com/users/pliljenberg/events{/privacy}","received_events_url":"https://api.github.com/users/pliljenberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":59,"created_at":"2015-07-30T11:46:07Z","updated_at":"2015-12-14T20:24:59Z","closed_at":null,"body":"We're running docker-registry-v2 on an AWS EC2 instance - backed by an EBS volume (switched from S3 since we thought that might be the underlying issue).\r\nFrom another AWS EC2 instance we're running a Bamboo CI agent which are building docker images and pushes them to our docker-registry.\r\nSeveral times each day we get failed builds caused by `docker push` getting `connection reset by peer`.\r\n  \r\n\r\n    [info] time=\"2015-07-30T10:00:31+02:00\" level=fatal msg=\"Error pushing to registry: Put https://docker-registry.**.**/v2/userhq/blobs/uploads/e60ca766-eb26-4adf-8cdb-fe7a127e3e4c?_state=a7vvRPLZLleaqCBw4xGmaxJZ-Z0Jc0SsFUUtcrnlqft7Ik5hbWUiOiJ1c2VyaHEiLCJVVUlEIjoiZTYwY2E3NjYtZWIyNi00YWRmLThjZGItZmU3YTEyN2UzZTRjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE1LTA3LTMwVDA4OjAwOjI2LjA0MzcwNzk2WiJ9&digest=sha256%3A1407c3b1319f21131f9da23c859ac406d2ae1051190611046c1666fc86dc5376: read tcp 1.2.3.4:443: connection reset by peer\" \r\n\r\nConfiguration and host information below, how can I debug this issue?\r\n\r\nThe docker registry is fronted by `nginx` configured as below:\r\n\r\n    user  nginx;\r\n    worker_processes  1;\r\n\r\n    events {\r\n      worker_connections 1024;\r\n      use epoll;\r\n      multi_accept on;\r\n    }\r\n\r\n    error_log  /var/log/nginx/error.log warn;\r\n    pid        /var/run/nginx.pid;\r\n\r\n    http {\r\n        include               /etc/nginx/mime.types;\r\n        default_type          application/octet-stream;\r\n\r\n        log_format            main  '$remote_addr - $remote_user [$time_local] \"$request\" '\r\n                                '$status $body_bytes_sent \"$http_referer\" '\r\n                                '\"$http_user_agent\" \"$http_x_forwarded_for\"';\r\n\r\n        access_log            /var/log/nginx/access.log  main;\r\n\r\n        sendfile              on;\r\n        keepalive_timeout     65;\r\n        ssl_session_timeout   10m;\r\n\r\n        ssl_certificate       /etc/nginx/ssl/server.crt;\r\n        ssl_certificate_key   /etc/nginx/ssl/server.key;\r\n\r\n        ssl_ciphers                 \"EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH\";\r\n        ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;\r\n        ssl_prefer_server_ciphers   on;\r\n        ssl_session_cache           shared:SSL:10m;\r\n        add_header                  Strict-Transport-Security \"max-age=63072000; includeSubdomains; preload\";\r\n        add_header                  X-Frame-Options DENY;\r\n        add_header                  X-Content-Type-Options nosniff;\r\n        ssl_session_tickets         off;\r\n        ssl_stapling                on; \r\n        ssl_stapling_verify         on;\r\n        \r\n        server {\r\n            listen 443 ssl;\r\n            server_name myregistrydomain.com;\r\n            \r\n            client_max_body_size 0;\r\n\r\n            chunked_transfer_encoding on;\r\n\r\n            client_body_buffer_size 100m;\r\n            \r\n            location /v2/ {\r\n                # Do not allow connections from docker 1.5 and earlier\r\n                # docker pre-1.6.0 did not properly set the user agent on ping, catch \"Go *\" user agents\r\n                if ($http_user_agent ~ \"^(docker\\/1\\.(3|4|5(?!\\.[0-9]-dev))|Go ).*$\" ) {\r\n                    return 404;\r\n                }\r\n\r\n                auth_basic \"registry.localhost\";\r\n                auth_basic_user_file /etc/nginx/registry.htpasswd;\r\n                add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;\r\n\r\n                proxy_pass                          http://docker-registry:5000;\r\n                proxy_set_header  Host              $http_host;   # required for docker client's sake\r\n                proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP\r\n                proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;\r\n                proxy_set_header  X-Forwarded-Proto $scheme;\r\n                proxy_read_timeout                  900;\r\n            }\r\n        }\r\n    }\r\n\r\nDocker registry configuration:\r\n\r\n    version: 0.1\r\n    log:\r\n      level: info\r\n      formatter: text\r\n      fields:\r\n        service: registry\r\n        environment: production\r\n    storage:\r\n      filesystem:\r\n        rootdirectory: /var/\r\n      cache:\r\n        layerinfo: redis\r\n      maintenance:\r\n        uploadpurging:\r\n          enabled: true\r\n          age: 72h\r\n          interval: 8h\r\n          dryrun: false\r\n    reporting:\r\n      newrelic:\r\n        licensekey: xxxxx \r\n        name: docker-registry\r\n        verbose: false\r\n    http:\r\n      addr: :5000\r\n    redis:\r\n      addr: cache:6379\r\n      db: 0\r\n      dialtimeout: 10ms\r\n      readtimeout: 10ms\r\n      writetimeout: 10ms\r\n      pool:\r\n        maxidle: 16\r\n        maxactive: 64\r\n        idletimeout: 300s\r\n\r\ndocker compose is used to start it all:\r\n\r\n    cache: \r\n      image: redis \r\n\r\n    nginx:\r\n      build: ./nginx\r\n      links: \r\n        - registry:docker-registry\r\n      ports: \r\n        - 443:443\r\n\r\n    registry: \r\n      build: ./registry\r\n      volumes: \r\n        - /data/docker:/var/docker\r\n      links: \r\n        - cache\r\n\r\n\r\nDocker registry host:\r\n   \r\n    $ docker info\r\n    Containers: 3\r\n    Images: 73\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 79\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.13.0-58-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 1\r\n    Total Memory: 3.676 GiB\r\n    Name: ip-172-31-24-143\r\n    ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA\r\n    WARNING: No swap limit support\r\n    ubuntu@ip-172-31-24-143:~$ docker info\r\n    Containers: 3\r\n    Images: 73\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 79\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.13.0-58-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 1\r\n    Total Memory: 3.676 GiB\r\n    Name: ip-172-31-24-143\r\n    ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA\r\n    WARNING: No swap limit support\r\n\r\n    $ docker version\r\n    Client version: 1.7.1\r\n    Client API version: 1.19\r\n    Go version (client): go1.4.2\r\n    Git commit (client): 786b29d\r\n    OS/Arch (client): linux/amd64\r\n    Server version: 1.7.1\r\n    Server API version: 1.19\r\n    Go version (server): go1.4.2\r\n    Git commit (server): 786b29d\r\n    OS/Arch (server): linux/amd64\r\n\r\nBamboo CI host:\r\n\r\n    $ docker info\r\n    Containers: 0\r\n    Images: 106\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 106\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Kernel Version: 3.13.0-52-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 8\r\n    Total Memory: 14.69 GiB\r\n    Name: ip-172-31-9-199\r\n    ID: ZTUG:L4YS:6JUT:KLH4:4PPE:5CGO:GBHD:4UI4:IA6F:OI3B:T5GY:7DX2\r\n    WARNING: No swap limit support\r\n\r\n    $ docker version\r\n    Client version: 1.6.2\r\n    Client API version: 1.18\r\n    Go version (client): go1.4.2\r\n    Git commit (client): 7c8fca2\r\n    OS/Arch (client): linux/amd64\r\n    Server version: 1.6.2\r\n    Server API version: 1.18\r\n    Go version (server): go1.4.2\r\n    Git commit (server): 7c8fca2\r\n    OS/Arch (server): linux/amd64"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/164549032","html_url":"https://github.com/docker/distribution/issues/785#issuecomment-164549032","issue_url":"https://api.github.com/repos/docker/distribution/issues/785","id":164549032,"user":{"login":"stevvooe","id":120601,"avatar_url":"https://avatars.githubusercontent.com/u/120601?v=3","gravatar_id":"","url":"https://api.github.com/users/stevvooe","html_url":"https://github.com/stevvooe","followers_url":"https://api.github.com/users/stevvooe/followers","following_url":"https://api.github.com/users/stevvooe/following{/other_user}","gists_url":"https://api.github.com/users/stevvooe/gists{/gist_id}","starred_url":"https://api.github.com/users/stevvooe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevvooe/subscriptions","organizations_url":"https://api.github.com/users/stevvooe/orgs","repos_url":"https://api.github.com/users/stevvooe/repos","events_url":"https://api.github.com/users/stevvooe/events{/privacy}","received_events_url":"https://api.github.com/users/stevvooe/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T20:24:59Z","updated_at":"2015-12-14T20:24:59Z","body":"@13josh Users experiencing this issue are having connection resets while pushing and pulling over some VPC configuration. If you are experiencing this issue, changing the buffer size may help, for certain configurations, as it can change the connection envelope for a single given chunk request. Albeit, we am not sure that changing buffer size having an effect is indicative of this problem. In fact, if that worked, it may be a different issue.\r\n\r\n@EliasGabrielsson @13josh I'd recommend relating your experience to https://forums.aws.amazon.com/thread.jspa?threadID=215090."}}