{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker-registry/issues/956","labels_url":"https://api.github.com/repos/docker/docker-registry/issues/956/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker-registry/issues/956/comments","events_url":"https://api.github.com/repos/docker/docker-registry/issues/956/events","html_url":"https://github.com/docker/docker-registry/issues/956","id":60168082,"number":956,"title":"docker client times out pulling large fs layer from S3-backed registry","user":{"login":"chaiken-verticloud","id":1719839,"avatar_url":"https://avatars.githubusercontent.com/u/1719839?v=3","gravatar_id":"","url":"https://api.github.com/users/chaiken-verticloud","html_url":"https://github.com/chaiken-verticloud","followers_url":"https://api.github.com/users/chaiken-verticloud/followers","following_url":"https://api.github.com/users/chaiken-verticloud/following{/other_user}","gists_url":"https://api.github.com/users/chaiken-verticloud/gists{/gist_id}","starred_url":"https://api.github.com/users/chaiken-verticloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chaiken-verticloud/subscriptions","organizations_url":"https://api.github.com/users/chaiken-verticloud/orgs","repos_url":"https://api.github.com/users/chaiken-verticloud/repos","events_url":"https://api.github.com/users/chaiken-verticloud/events{/privacy}","received_events_url":"https://api.github.com/users/chaiken-verticloud/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-06T22:28:04Z","updated_at":"2015-03-06T23:30:39Z","closed_at":null,"body":"At Altiscale, we have been running a standalone docker registry backed by S3.  The registry works fine with STORAGE_REDIRECT=true.  We're tightening up registry security and bandwidth usage by setting STORAGE_REDIRECT=false, but our docker clients now time out when fetching large fs layers.\r\n\r\nHere's the scenario:\r\n- On the docker client, pull the image with the large fs layer...\r\n```\r\n[docker_client_host]$ docker pull registry reghost.altiscale.com/container_name\r\nPulling repository reghost.altiscale.com/container_name\r\n921a3438ca3d: Pulling image (latest) from reghost.altiscale.com/container_name\r\n921a3438ca3d: Pulling image (latest) from reghost.altiscale.com/container_name, endpoint: https://reghost.altiscale.com/v1/ \r\n921a3438ca3d: Pulling metadata \r\n921a3438ca3d: Pulling fs layer \r\n```\r\n- The docker registry receives a request for the layer.\r\n- This code in ```depends/docker-registry-core/docker_registry/core/boto.py:Base.stream_read``` decides to fetch the layer using the ParallelKey class.\r\n```python\r\n        if not bytes_range and key.size > 1024 * 1024:\r\n            # Use the parallel key only if the key size is > 1MB                                                                                                                  \r\n            # And if bytes_range is not enabled (since ParallelKey is already                                                                                                     \r\n            # using bytes range)                                                                                                                                                  \r\n            key = ParallelKey(key)\r\n```\r\n- This code in ```depends/docker-registry-core/docker_registry/core/boto.py:ParallelKey.__init__``` opens a temporary file:\r\n```python\r\nself._tmpfile = tempfile.NamedTemporaryFile(mode='rb')\r\n```\r\n- In 5 (CONCURRENCY) threads, this code in ```depends/docker-registry-core/docker_registry/core/boto.py:ParallelKey._fetch_part``` starts transferring 1/5 of the object at the corresponding offset in the temporary file:\r\n```python\r\nwith open(fname, 'wb') as f:\r\n            f.seek(min_cur)\r\n            brange = 'bytes={0}-{1}'.format(min_cur, max_cur)\r\n            boto_key.get_contents_to_file(f, headers={'Range': brange})\r\n            boto_key.close()\r\n```\r\n- The fs layer object in S3 is so large that the docker client times out while these 5 threads are running, and retries.\r\n- The above process repeats: opens a second temporary file, spawns 5 threads, and the 5 new threads start transferring the object into the second temporary file.  At this point, 10 threads are running, transferring into two different temporary files.\r\n- Eventually one of the first 5 threads completes the call to ```boto_key.get_contents_to_file``` and calls ```ParallelKey._refresh_max_completed_byte```, which emits the message ```buffering complete at 20.0% of the total transfer; now serving straight from the tempfile```.  At this point, the client has already timed out on its original request, so the initial fetch of the S3 object into the temporary file was not useful work.\r\n- This process repeats: each time the threads complete transferring the object, the docker client has already timed out.\r\n- After two tries, the docker client emits this error after all of its retries fail:\r\n```\r\n921a3438ca3d: Error pulling image (latest) from reghost.altiscale.com/container_name, Get https://reghost.altiscale.com/v1/images/921a3438ca3d58320b54f1117a992b5ce2c2f7305ddb354250364a19fb3d37e6/layer: read tcp ip_address_of_reghost:443: i/o timeout \r\n```\r\n\r\nI have confirmed this same behavior using curl directly on the docker registry host.  This curl command demonstrates the same behavior, except that curl doesn't time out so that the initial download eventually works.  When running the curl command, curl does not download the first byte of the file until the first of the 5 docker registry threads calls ```ParallelKey._refresh_max_completed_byte```.  Once the first thread completes, curl starts downloading the layer.\r\n```\r\n[reghost.altiscale.com]$ curl -v -O http://localhost:5000/v1/images/921a3438ca3d58320b54f1117a992b5ce2c2f7305ddb354250364a19fb3d37e6/layer\r\n```\r\nThis curl command uses the Range header to disable ParallelKey.  This header makes the ```if not bytes_range``` conditional in the code above false, so that ParallelKey does not run.  In this case, curl starts downloading bytes immediately.\r\n```\r\n[reghost.altiscale.com]$  curl -v -O -H 'Range: bytes=0-' http://localhost:5000/v1/images/921a3438ca3d58320b54f1117a992b5ce2c2f7305ddb354250364a19fb3d37e6/layer\r\n```\r\n\r\nHere are some potential solutions for the problem that I would be happy to contribute as code:\r\n1. Modify ```boto.py:ParallelKey._fetch_part``` so that it limits the size of any individual transfer,  and therefore calls ```ParallelKey._refresh_max_completed_byte``` faster.  This change would be pretty easy, but each thread would still work on its own segment of the file.  The result is that the streaming transfer of the client would be rate limited by single thread transfer performance.\r\n2. Modify ```boto.py:ParallelKey``` so that each thread picks up a new range of bytes to transfer as close to the beginning of the object as possible, but not overlapping with the ranges being transferred by other threads.  This change would be more substantial (and harder to get right), but the all of the threads would contribute to the rate that the client would be able to transfer.\r\n3. Modify the docker-registry configuration code to allow the operator to disable ```ParallelKey``` entirely, or if an S3 object is over a configured size.\r\n4. Remove the ```ParallelKey``` code from ```boto.py```.\r\n5. Provide the ability to configure the docker client with a longer timeout.  This approach would eliminate the need to modify docker-registry code, but does not improve performance for the client by allowing it to do a streaming transfer in parallel with the docker registry's transfer from S3.\r\n6. Provide the ability for the docker client to disable ```ParallelKey``` by passing the Range HTTP header to the docker registry.  This approach would work with the current docker registry code, but seems like it's too much of a hack because it depends on the internal implementation of ```boto.py:Base.stream_read```.\r\n\r\nAny thoughts on the best way to proceed?\r\n\r\n...or maybe I missed some obvious way to fix this issue?  Please let me know if there is some obvious way to get a standalone registry working with STORAGE_REDIRECT=false and large fs layers stored as S3 objects!\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker-registry/issues/comments/77655547","html_url":"https://github.com/docker/docker-registry/issues/956#issuecomment-77655547","issue_url":"https://api.github.com/repos/docker/docker-registry/issues/956","id":77655547,"user":{"login":"dmp42","id":332946,"avatar_url":"https://avatars.githubusercontent.com/u/332946?v=3","gravatar_id":"","url":"https://api.github.com/users/dmp42","html_url":"https://github.com/dmp42","followers_url":"https://api.github.com/users/dmp42/followers","following_url":"https://api.github.com/users/dmp42/following{/other_user}","gists_url":"https://api.github.com/users/dmp42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmp42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmp42/subscriptions","organizations_url":"https://api.github.com/users/dmp42/orgs","repos_url":"https://api.github.com/users/dmp42/repos","events_url":"https://api.github.com/users/dmp42/events{/privacy}","received_events_url":"https://api.github.com/users/dmp42/received_events","type":"User","site_admin":false},"created_at":"2015-03-06T23:30:39Z","updated_at":"2015-03-06T23:30:39Z","body":"Ok, that sounds bad enough.\r\n\r\nGiven the effort is now on https://github.com/docker/distribution (new golang implementation), little effort is going to be put in the python-registry here.\r\n\r\nMy suggestion is either remove the parallel key code entirely, or better, make it configurable (and disable it by default).\r\n\r\nIf you are willing to PR this, that would be awesome..."}}