{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker-registry/issues/540","labels_url":"https://api.github.com/repos/docker/docker-registry/issues/540/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker-registry/issues/540/comments","events_url":"https://api.github.com/repos/docker/docker-registry/issues/540/events","html_url":"https://github.com/docker/docker-registry/issues/540","id":41252037,"number":540,"title":"Registry degrades and reports corruption","user":{"login":"Henkis","id":234481,"avatar_url":"https://avatars.githubusercontent.com/u/234481?v=3","gravatar_id":"","url":"https://api.github.com/users/Henkis","html_url":"https://github.com/Henkis","followers_url":"https://api.github.com/users/Henkis/followers","following_url":"https://api.github.com/users/Henkis/following{/other_user}","gists_url":"https://api.github.com/users/Henkis/gists{/gist_id}","starred_url":"https://api.github.com/users/Henkis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Henkis/subscriptions","organizations_url":"https://api.github.com/users/Henkis/orgs","repos_url":"https://api.github.com/users/Henkis/repos","events_url":"https://api.github.com/users/Henkis/events{/privacy}","received_events_url":"https://api.github.com/users/Henkis/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker-registry/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/docker/docker-registry/labels/hardening","name":"hardening","color":"0052cc"}],"state":"open","locked":false,"assignee":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/docker/docker-registry/milestones/7","html_url":"https://github.com/docker/docker-registry/milestones/1.0","labels_url":"https://api.github.com/repos/docker/docker-registry/milestones/7/labels","id":739530,"number":7,"title":"1.0","description":"Last release for the v1 docker registry.\r\nWill be maintenance only, focusing on cleanup, bugfixing, and documentation enhancements.","creator":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":35,"state":"open","created_at":"2014-08-01T09:59:48Z","updated_at":"2015-01-22T21:58:17Z","due_on":"2015-02-09T08:00:00Z","closed_at":null},"comments":27,"created_at":"2014-08-27T06:16:55Z","updated_at":"2015-03-17T01:14:12Z","closed_at":null,"body":"We are using a privately hosted registry on amazon which seems to degrade after we have pulled larger images (1GB+) a few times from it, it finely stop responding. A restart of the registry container seems to fix the problem for a while:\r\n\r\n```\r\ndf29cb2b16e8: Download complete\r\n8673c2be5440: Download complete\r\nefb1cc4f166f: Error downloading dependent layers\r\n2014/08/26 23:09:14 Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/server, flate: corrupt input before offset 13576289\r\n```\r\n\r\nAnd again:\r\n\r\n```\r\nPulling repository 256.31.14.183/server\r\na44fa854766a: Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/adserver, archive/tar: invalid tar header 1/, archive/tar: invalid tar header\r\n5e66087f3ffe: Error downloading dependent layers\r\n2014/08/27 06:03:14 Error pulling image (97a9cc4076fd69d8a35be028a74f0e00b8066555) from 256.31.14.183/server, archive/tar: invalid tar header\r\n```\r\n\r\nWe often see these messages in the log but not always:\r\n\r\n```\r\n<Greenlet at 0x7f8bbb7269b0: <bound method ParallelKey._fetch_part of <docker_registry.core.boto.ParallelKey object at 0x7f8bbb2be550>>('/tmp/tmpPPltfO', 1, 13495293, 26990585)> failed with timeout\r\n```\r\n\r\nWe currently use the latest registry (registry 0.8.0 cd3581c06bdc) but have hade the same problems earlier.\r\n\r\nRegistry config:\r\n\r\nDockerfile:\r\n```\r\nFROM registry\r\nADD ./config.yml /docker-registry/config/config.yml\r\n```\r\n\r\nconfig.yml:\r\n```\r\ncommon:\r\n    loglevel: info\r\n    secret_key: <OURLITTLESECRET>\r\n\r\n\r\n# This is the default configuration when no flavor is specified\r\ndev:\r\n    storage: local\r\n    storage_path: /tmp/registry\r\n    loglevel: debug\r\n\r\n# To specify another flavor, set the environment variable SETTINGS_FLAVOR\r\n# $ export SETTINGS_FLAVOR=prod\r\nprod:\r\n    storage: s3\r\n    storage_path: /prod\r\n    loglevel: debug\r\n    # Amazon S3 Storage Configuration\r\n    s3_access_key: <OURLITTLESECRET>\r\n    s3_secret_key: <OURLITTLESECRET>\r\n    s3_bucket: arkivdigital-docker-registry\r\n    boto_bucket: ourrepo-registry\r\n    # Enabling LRU cache for small files. This speeds up read/write on small files\r\n    # when using a remote storage backend (like S3).\r\n    cache:\r\n        host: docker-registryblabla.cache.amazonaws.com\r\n        port: 6379\r\n        db: 0\r\n    cache_lru:\r\n        host: docker-registryblabla.cache.amazonaws.com\r\n        port: 6379\r\n        db: 0\r\n\r\n    standalone: true\r\n    disable_token_auth: true\r\n\r\n# This flavor is automatically used by unit tests\r\ntest:\r\n    storage: local\r\n    storage_path: /tmp/test\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker-registry/issues/comments/82016346","html_url":"https://github.com/docker/docker-registry/issues/540#issuecomment-82016346","issue_url":"https://api.github.com/repos/docker/docker-registry/issues/540","id":82016346,"user":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"created_at":"2015-03-17T01:14:12Z","updated_at":"2015-03-17T01:14:12Z","body":"I would definitely recommend using the Redis LRU cache.\r\nI would also advise to delegate actual bits delivery to either S3 or a Cloudfront view of it (https://github.com/docker/docker-registry/blob/master/ADVANCED.md#s3-storage-with-cloudfront)"}}