{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker-registry/issues/540","labels_url":"https://api.github.com/repos/docker/docker-registry/issues/540/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker-registry/issues/540/comments","events_url":"https://api.github.com/repos/docker/docker-registry/issues/540/events","html_url":"https://github.com/docker/docker-registry/issues/540","id":41252037,"number":540,"title":"Registry degrades and reports corruption","user":{"login":"Henkis","id":234481,"avatar_url":"https://avatars.githubusercontent.com/u/234481?v=3","gravatar_id":"","url":"https://api.github.com/users/Henkis","html_url":"https://github.com/Henkis","followers_url":"https://api.github.com/users/Henkis/followers","following_url":"https://api.github.com/users/Henkis/following{/other_user}","gists_url":"https://api.github.com/users/Henkis/gists{/gist_id}","starred_url":"https://api.github.com/users/Henkis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Henkis/subscriptions","organizations_url":"https://api.github.com/users/Henkis/orgs","repos_url":"https://api.github.com/users/Henkis/repos","events_url":"https://api.github.com/users/Henkis/events{/privacy}","received_events_url":"https://api.github.com/users/Henkis/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker-registry/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/docker/docker-registry/labels/hardening","name":"hardening","color":"0052cc"}],"state":"open","locked":false,"assignee":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/docker/docker-registry/milestones/7","html_url":"https://github.com/docker/docker-registry/milestones/1.0","labels_url":"https://api.github.com/repos/docker/docker-registry/milestones/7/labels","id":739530,"number":7,"title":"1.0","description":"Last release for the v1 docker registry.\r\nWill be maintenance only, focusing on cleanup, bugfixing, and documentation enhancements.","creator":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":35,"state":"open","created_at":"2014-08-01T09:59:48Z","updated_at":"2015-01-22T21:58:17Z","due_on":"2015-02-09T08:00:00Z","closed_at":null},"comments":21,"created_at":"2014-08-27T06:16:55Z","updated_at":"2015-03-10T21:46:13Z","closed_at":null,"body":"We are using a privately hosted registry on amazon which seems to degrade after we have pulled larger images (1GB+) a few times from it, it finely stop responding. A restart of the registry container seems to fix the problem for a while:\r\n\r\n```\r\ndf29cb2b16e8: Download complete\r\n8673c2be5440: Download complete\r\nefb1cc4f166f: Error downloading dependent layers\r\n2014/08/26 23:09:14 Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/server, flate: corrupt input before offset 13576289\r\n```\r\n\r\nAnd again:\r\n\r\n```\r\nPulling repository 256.31.14.183/server\r\na44fa854766a: Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/adserver, archive/tar: invalid tar header 1/, archive/tar: invalid tar header\r\n5e66087f3ffe: Error downloading dependent layers\r\n2014/08/27 06:03:14 Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/server, archive/tar: invalid tar header\r\n```\r\n\r\nWe often see these messages in the log but not always:\r\n\r\n```\r\n<Greenlet at 0x7f8bbb7269b0: <bound method ParallelKey._fetch_part of <docker_registry.core.boto.ParallelKey object at 0x7f8bbb2be550>>('/tmp/tmpPPltfO', 1, 13495293, 26990585)> failed with timeout\r\n```\r\n\r\nWe currently use the latest registry (registry 0.8.0 cd3581c06bdc) but have hade the same problems earlier.\r\n\r\nRegistry config:\r\n\r\nDockerfile:\r\n```\r\nFROM registry\r\nADD ./config.yml /docker-registry/config/config.yml\r\n```\r\n\r\nconfig.yml:\r\n```\r\ncommon:\r\n    loglevel: info\r\n    secret_key: <OURLITTLESECRET>\r\n\r\n\r\n# This is the default configuration when no flavor is specified\r\ndev:\r\n    storage: local\r\n    storage_path: /tmp/registry\r\n    loglevel: debug\r\n\r\n# To specify another flavor, set the environment variable SETTINGS_FLAVOR\r\n# $ export SETTINGS_FLAVOR=prod\r\nprod:\r\n    storage: s3\r\n    storage_path: /prod\r\n    loglevel: debug\r\n    # Amazon S3 Storage Configuration\r\n    s3_access_key: <OURLITTLESECRET>\r\n    s3_secret_key: <OURLITTLESECRET>\r\n    s3_bucket: arkivdigital-docker-registry\r\n    boto_bucket: ourrepo-registry\r\n    # Enabling LRU cache for small files. This speeds up read/write on small files\r\n    # when using a remote storage backend (like S3).\r\n    cache:\r\n        host: docker-registryblabla.cache.amazonaws.com\r\n        port: 6379\r\n        db: 0\r\n    cache_lru:\r\n        host: docker-registryblabla.cache.amazonaws.com\r\n        port: 6379\r\n        db: 0\r\n\r\n    standalone: true\r\n    disable_token_auth: true\r\n\r\n# This flavor is automatically used by unit tests\r\ntest:\r\n    storage: local\r\n    storage_path: /tmp/test\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker-registry/issues/comments/78155400","html_url":"https://github.com/docker/docker-registry/issues/540#issuecomment-78155400","issue_url":"https://api.github.com/repos/docker/docker-registry/issues/540","id":78155400,"user":{"login":"dsw88","id":3180496,"avatar_url":"https://avatars.githubusercontent.com/u/3180496?v=3","gravatar_id":"","url":"https://api.github.com/users/dsw88","html_url":"https://github.com/dsw88","followers_url":"https://api.github.com/users/dsw88/followers","following_url":"https://api.github.com/users/dsw88/following{/other_user}","gists_url":"https://api.github.com/users/dsw88/gists{/gist_id}","starred_url":"https://api.github.com/users/dsw88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsw88/subscriptions","organizations_url":"https://api.github.com/users/dsw88/orgs","repos_url":"https://api.github.com/users/dsw88/repos","events_url":"https://api.github.com/users/dsw88/events{/privacy}","received_events_url":"https://api.github.com/users/dsw88/received_events","type":"User","site_admin":false},"created_at":"2015-03-10T21:46:13Z","updated_at":"2015-03-10T21:46:13Z","body":"We've got this issue too on our registries. We're running them inside an AWS VPC. We're running one registry container per host, with 6 instance currently. Those instances are fronted by an elastic load balancer whose timeout it set to 5 mins. We're using S3 as a storage backend.\r\n\r\nAfter finding this thread, we set the registry with the -e STORAGE_REDIRECT=true option to delegate image downloads directly to S3. That has helped tremendously with the myriad of errors we were getting such as EOF.\r\n\r\nHowever, we're still getting EOF errors on some registry calls that aren't actual image layer retrievals. For example, I saw an EOF error on a /ancestry call. I don't see any errors in the Docker Registry logs when these types of issues happen so I'm sort of at a loss.\r\n\r\nThese errors seem to happen when we have a heavier load on the registries such as 5-10 images pulling from the registries from the same time. However, that doesn't seem like a heavy load especially with storage redirect and 6 instances of the registry behind a load balancer."}}