{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/785","labels_url":"https://api.github.com/repos/docker/distribution/issues/785/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/785/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/785/events","html_url":"https://github.com/docker/distribution/issues/785","id":98148068,"number":785,"title":"Intermittent \"connection reset by peer\" while pushing image","user":{"login":"pliljenberg","id":378483,"avatar_url":"https://avatars.githubusercontent.com/u/378483?v=3","gravatar_id":"","url":"https://api.github.com/users/pliljenberg","html_url":"https://github.com/pliljenberg","followers_url":"https://api.github.com/users/pliljenberg/followers","following_url":"https://api.github.com/users/pliljenberg/following{/other_user}","gists_url":"https://api.github.com/users/pliljenberg/gists{/gist_id}","starred_url":"https://api.github.com/users/pliljenberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pliljenberg/subscriptions","organizations_url":"https://api.github.com/users/pliljenberg/orgs","repos_url":"https://api.github.com/users/pliljenberg/repos","events_url":"https://api.github.com/users/pliljenberg/events{/privacy}","received_events_url":"https://api.github.com/users/pliljenberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":42,"created_at":"2015-07-30T11:46:07Z","updated_at":"2015-10-21T23:06:39Z","closed_at":null,"body":"We're running docker-registry-v2 on an AWS EC2 instance - backed by an EBS volume (switched from S3 since we thought that might be the underlying issue).\r\nFrom another AWS EC2 instance we're running a Bamboo CI agent which are building docker images and pushes them to our docker-registry.\r\nSeveral times each day we get failed builds caused by `docker push` getting `connection reset by peer`.\r\n  \r\n\r\n    [info] time=\"2015-07-30T10:00:31+02:00\" level=fatal msg=\"Error pushing to registry: Put https://docker-registry.**.**/v2/userhq/blobs/uploads/e60ca766-eb26-4adf-8cdb-fe7a127e3e4c?_state=a7vvRPLZLleaqCBw4xGmaxJZ-Z0Jc0SsFUUtcrnlqft7Ik5hbWUiOiJ1c2VyaHEiLCJVVUlEIjoiZTYwY2E3NjYtZWIyNi00YWRmLThjZGItZmU3YTEyN2UzZTRjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE1LTA3LTMwVDA4OjAwOjI2LjA0MzcwNzk2WiJ9&digest=sha256%3A1407c3b1319f21131f9da23c859ac406d2ae1051190611046c1666fc86dc5376: read tcp 1.2.3.4:443: connection reset by peer\" \r\n\r\nConfiguration and host information below, how can I debug this issue?\r\n\r\nThe docker registry is fronted by `nginx` configured as below:\r\n\r\n    user  nginx;\r\n    worker_processes  1;\r\n\r\n    events {\r\n      worker_connections 1024;\r\n      use epoll;\r\n      multi_accept on;\r\n    }\r\n\r\n    error_log  /var/log/nginx/error.log warn;\r\n    pid        /var/run/nginx.pid;\r\n\r\n    http {\r\n        include               /etc/nginx/mime.types;\r\n        default_type          application/octet-stream;\r\n\r\n        log_format            main  '$remote_addr - $remote_user [$time_local] \"$request\" '\r\n                                '$status $body_bytes_sent \"$http_referer\" '\r\n                                '\"$http_user_agent\" \"$http_x_forwarded_for\"';\r\n\r\n        access_log            /var/log/nginx/access.log  main;\r\n\r\n        sendfile              on;\r\n        keepalive_timeout     65;\r\n        ssl_session_timeout   10m;\r\n\r\n        ssl_certificate       /etc/nginx/ssl/server.crt;\r\n        ssl_certificate_key   /etc/nginx/ssl/server.key;\r\n\r\n        ssl_ciphers                 \"EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH\";\r\n        ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;\r\n        ssl_prefer_server_ciphers   on;\r\n        ssl_session_cache           shared:SSL:10m;\r\n        add_header                  Strict-Transport-Security \"max-age=63072000; includeSubdomains; preload\";\r\n        add_header                  X-Frame-Options DENY;\r\n        add_header                  X-Content-Type-Options nosniff;\r\n        ssl_session_tickets         off;\r\n        ssl_stapling                on; \r\n        ssl_stapling_verify         on;\r\n        \r\n        server {\r\n            listen 443 ssl;\r\n            server_name myregistrydomain.com;\r\n            \r\n            client_max_body_size 0;\r\n\r\n            chunked_transfer_encoding on;\r\n\r\n            client_body_buffer_size 100m;\r\n            \r\n            location /v2/ {\r\n                # Do not allow connections from docker 1.5 and earlier\r\n                # docker pre-1.6.0 did not properly set the user agent on ping, catch \"Go *\" user agents\r\n                if ($http_user_agent ~ \"^(docker\\/1\\.(3|4|5(?!\\.[0-9]-dev))|Go ).*$\" ) {\r\n                    return 404;\r\n                }\r\n\r\n                auth_basic \"registry.localhost\";\r\n                auth_basic_user_file /etc/nginx/registry.htpasswd;\r\n                add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;\r\n\r\n                proxy_pass                          http://docker-registry:5000;\r\n                proxy_set_header  Host              $http_host;   # required for docker client's sake\r\n                proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP\r\n                proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;\r\n                proxy_set_header  X-Forwarded-Proto $scheme;\r\n                proxy_read_timeout                  900;\r\n            }\r\n        }\r\n    }\r\n\r\nDocker registry configuration:\r\n\r\n    version: 0.1\r\n    log:\r\n      level: info\r\n      formatter: text\r\n      fields:\r\n        service: registry\r\n        environment: production\r\n    storage:\r\n      filesystem:\r\n        rootdirectory: /var/\r\n      cache:\r\n        layerinfo: redis\r\n      maintenance:\r\n        uploadpurging:\r\n          enabled: true\r\n          age: 72h\r\n          interval: 8h\r\n          dryrun: false\r\n    reporting:\r\n      newrelic:\r\n        licensekey: xxxxx \r\n        name: docker-registry\r\n        verbose: false\r\n    http:\r\n      addr: :5000\r\n    redis:\r\n      addr: cache:6379\r\n      db: 0\r\n      dialtimeout: 10ms\r\n      readtimeout: 10ms\r\n      writetimeout: 10ms\r\n      pool:\r\n        maxidle: 16\r\n        maxactive: 64\r\n        idletimeout: 300s\r\n\r\ndocker compose is used to start it all:\r\n\r\n    cache: \r\n      image: redis \r\n\r\n    nginx:\r\n      build: ./nginx\r\n      links: \r\n        - registry:docker-registry\r\n      ports: \r\n        - 443:443\r\n\r\n    registry: \r\n      build: ./registry\r\n      volumes: \r\n        - /data/docker:/var/docker\r\n      links: \r\n        - cache\r\n\r\n\r\nDocker registry host:\r\n   \r\n    $ docker info\r\n    Containers: 3\r\n    Images: 73\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 79\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.13.0-58-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 1\r\n    Total Memory: 3.676 GiB\r\n    Name: ip-172-31-24-143\r\n    ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA\r\n    WARNING: No swap limit support\r\n    ubuntu@ip-172-31-24-143:~$ docker info\r\n    Containers: 3\r\n    Images: 73\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 79\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.13.0-58-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 1\r\n    Total Memory: 3.676 GiB\r\n    Name: ip-172-31-24-143\r\n    ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA\r\n    WARNING: No swap limit support\r\n\r\n    $ docker version\r\n    Client version: 1.7.1\r\n    Client API version: 1.19\r\n    Go version (client): go1.4.2\r\n    Git commit (client): 786b29d\r\n    OS/Arch (client): linux/amd64\r\n    Server version: 1.7.1\r\n    Server API version: 1.19\r\n    Go version (server): go1.4.2\r\n    Git commit (server): 786b29d\r\n    OS/Arch (server): linux/amd64\r\n\r\nBamboo CI host:\r\n\r\n    $ docker info\r\n    Containers: 0\r\n    Images: 106\r\n    Storage Driver: aufs\r\n     Root Dir: /var/lib/docker/aufs\r\n     Backing Filesystem: extfs\r\n     Dirs: 106\r\n     Dirperm1 Supported: false\r\n    Execution Driver: native-0.2\r\n    Kernel Version: 3.13.0-52-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 8\r\n    Total Memory: 14.69 GiB\r\n    Name: ip-172-31-9-199\r\n    ID: ZTUG:L4YS:6JUT:KLH4:4PPE:5CGO:GBHD:4UI4:IA6F:OI3B:T5GY:7DX2\r\n    WARNING: No swap limit support\r\n\r\n    $ docker version\r\n    Client version: 1.6.2\r\n    Client API version: 1.18\r\n    Go version (client): go1.4.2\r\n    Git commit (client): 7c8fca2\r\n    OS/Arch (client): linux/amd64\r\n    Server version: 1.6.2\r\n    Server API version: 1.18\r\n    Go version (server): go1.4.2\r\n    Git commit (server): 7c8fca2\r\n    OS/Arch (server): linux/amd64"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/150049455","html_url":"https://github.com/docker/distribution/issues/785#issuecomment-150049455","issue_url":"https://api.github.com/repos/docker/distribution/issues/785","id":150049455,"user":{"login":"robertfirek","id":3847372,"avatar_url":"https://avatars.githubusercontent.com/u/3847372?v=3","gravatar_id":"","url":"https://api.github.com/users/robertfirek","html_url":"https://github.com/robertfirek","followers_url":"https://api.github.com/users/robertfirek/followers","following_url":"https://api.github.com/users/robertfirek/following{/other_user}","gists_url":"https://api.github.com/users/robertfirek/gists{/gist_id}","starred_url":"https://api.github.com/users/robertfirek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robertfirek/subscriptions","organizations_url":"https://api.github.com/users/robertfirek/orgs","repos_url":"https://api.github.com/users/robertfirek/repos","events_url":"https://api.github.com/users/robertfirek/events{/privacy}","received_events_url":"https://api.github.com/users/robertfirek/received_events","type":"User","site_admin":false},"created_at":"2015-10-21T23:06:39Z","updated_at":"2015-10-21T23:06:39Z","body":"@stevvooe Someone on https://forums.aws.amazon.com/thread.jspa?threadID=215090 asked for some instance to check. Does anyone have spare one?"}}