{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/459","labels_url":"https://api.github.com/repos/docker/distribution/issues/459/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/459/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/459/events","html_url":"https://github.com/docker/distribution/issues/459","id":71654920,"number":459,"title":"Registry V2 mirror proposal","user":{"login":"RichardScothern","id":922018,"avatar_url":"https://avatars.githubusercontent.com/u/922018?v=3","gravatar_id":"","url":"https://api.github.com/users/RichardScothern","html_url":"https://github.com/RichardScothern","followers_url":"https://api.github.com/users/RichardScothern/followers","following_url":"https://api.github.com/users/RichardScothern/following{/other_user}","gists_url":"https://api.github.com/users/RichardScothern/gists{/gist_id}","starred_url":"https://api.github.com/users/RichardScothern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RichardScothern/subscriptions","organizations_url":"https://api.github.com/users/RichardScothern/orgs","repos_url":"https://api.github.com/users/RichardScothern/repos","events_url":"https://api.github.com/users/RichardScothern/events{/privacy}","received_events_url":"https://api.github.com/users/RichardScothern/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/distribution/labels/docker/1.7","name":"docker/1.7","color":"eb6420"},{"url":"https://api.github.com/repos/docker/distribution/labels/enhancement","name":"enhancement","color":"bfe5bf"},{"url":"https://api.github.com/repos/docker/distribution/labels/proposal","name":"proposal","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2015-04-28T17:42:54Z","updated_at":"2015-06-29T16:28:14Z","closed_at":null,"body":"Milestones\r\n\r\n- [x] Client changes\r\n- [x] Cache-control headers\r\n- [x] Image and Manifest pull-through\r\n  - [ ] Serialized remote blob fetch\r\n  - [x] Redirect handling\r\n- [ ] Config\r\n- [ ] Cache expiration\r\n- [ ] Namespace integration / access control implementation\r\n- [ ] Tests\r\n- [ ] LGTM from @dmcgowan @stevvooe \r\n \r\nProposed for #19 \r\n\r\n# Registry V2 mirror proposal\r\n\r\nA registry should be able to operate as a pull-through mirroring cache. This means that if a pull operation cannot proceed locally due to missing content, the registry should be able to defer the request to an upstream registry.\r\n\r\nUnlike the V1 registry mirroring feature, a V2 registry mirror can proxy to a private registry.  This will allow more flexible registry topologies. \r\n\r\n\r\n## Behavior\r\nfrom [Issue #19](https://github.com/docker/distribution/issues/19):\r\n \r\nPull\r\n\r\nWhen pulling a manifest, if the content is available locally, it will be served as is. Optionally, the local may check the remote if the content has been updated, using conditional http requests and update the local content. If it is not available locally, the request should be forwarded to the remote registry. If the remote request proceeds, the manifest should be stored locally then served in response to the local request.\r\n\r\nWhen pulling a blob, if the content is available locally, serve it as is. If not available locally, forward the request to the remote registry. If the data is directly available, the data should be forwarded to the client and stored locally, concurrently. If the remote issues a redirect, the local registry should download the data into the local cache and serve the data directly.\r\n\r\nPush\r\n\r\nAll push operations are only attempted on the \"local\" registry. If they fail, they will not be forwarded to the remote registry.\r\n\r\n\r\n### Docker Daemon control\r\n\r\nAs with the v1 registry, the Docker daemon will supply the `--registry-mirror` option to make requests through the proxying registry.\r\n\r\n## Implementation\r\n\r\n### HTTP Cache Headers\r\n\r\n\r\n`Cache-Control`\r\n\r\nThe value in this header specifies how long the data associated with the request will be cached for.  The TTL will be obeyed by the registry mirror and retained with the response to the client in order to take advantage of intermediate web caches.  The TTL will be specified by the `max-age` directive.\r\n\r\n`If-None-Match`\r\n\r\nThe value of this header will be used by the registry mirror to check for updated images in the back-end registry.\r\n\r\n`Etag`\r\n\r\nAn Etag header will be used in combination with the If-None-Match header.  The value of the Etag header will be the same as `Docker-Content-Digest`\r\n\r\n\r\n### TTL & Flushing\r\n\r\nA registry mirror will obey TTLs specified for mutable content.  When the TTL for a given resource has expired it will be removed from the mirror and fetched from the upstream registry on the subsequent request.\r\n\r\nImmutable data does not have to obey TTL as it can never become invalidated and thus will remain in the cache. \r\n\r\n### Credentials\r\n\r\nA registry mirror will not forward the credentials from the client request.  A registry mirror will have its own credentials specified in the configuration.  A registry mirror will send an altered `User-Agent` HTTP header to identify itself as a mirror.  Identification of the original client can be forwarded by the mirror using the `X-Forwarded-For` HTTP header.\r\n\r\n### Registry client Behavior\r\n\r\nMuch of the client behavior will be defined in the registry client package (see dependencies).  However the following behavior will be important to ensure good cache performance.\r\n\r\n   * Conditional checks\r\n     \r\n     The local registry will avoid unnecessary data transfers by issuing conditional HTTP requests to the remote registry.  This is especially important for large layers.  In order to avoid unnecessary data transfer, an Etag HTTP  header (see above) will be used to check freshness.\r\n\r\n   * Concurrent layer pulling\r\n   \r\n      Where a set of blobs is known at request time, the individual layers will be downloaded to the mirror concurrently.  In the future, batching these requests will provide a greater performance gain.\r\n      \r\n \r\n### Access Control\r\n\r\nA registry mirror will be configurable with an access control list which will give fine grained control over which source images are served from.  \r\n\r\nAccess control will consists image names with wildcard cards (including tags), and the value will specify wether to only serve from the local or the upstream registry. \r\n\r\ne.g.\r\n\r\n```\r\nacl:\r\n   'library/*': \"remote\"\r\n   'mycorp/product': \"remote\"\r\n   'mycorp/product:testing': \"local\"\r\n\r\n```\r\n\r\n### Compression\r\n\r\nIn order to reduce bandwidth on a network, the Registry will support transport encoding.  The Registry mirror will set the HTTP `Accept Encoding` header with supported compression algorithms and the Registry backend will optionally use this to compress certain layers \r\n\r\n### Configuration\r\n\r\nThe following aspects of the Caching behavior will be supported in the config file.\r\n\r\nOption | Meaning\r\n------ | -------\r\nupstream  | URL of upstream registry\r\nacl | *see example above*\r\nUsername | Username to present to remote registry\r\nPassword | Password to present to remote registry\r\n\r\n### Integration with the Layer Cache\r\n\r\nThe mirror cache belonging to the registry mirror will be populated with data pulled from the remote registry.\r\n\r\n### Circular Dependencies\r\n\r\nAllowing proxying to arbitrary registry open up the possibility of circular dependencies.  Request sources can be tracked in a number of different ways: \r\n\r\n   * The number of proxy hops can be tracked and capped at an arbitrary number\r\n   * The proxies involved in a request can be tracked in an HTTP header (e.g. X-Forwarded-For) and the request terminated when a circle is detected. \r\n\r\n## Code Dependencies\r\n\r\nImplementation is dependent on some current pull requests.  A hard requirement is a stable registry client. \r\n\r\n  * [#387 Add client implementation of distribution interface](https://github.com/docker/distribution/pull/387)\r\n\r\nOther pull request which are preferable to reduce code churn include\r\nvarious refactor pull requests listed [https://github.com/docker/distribution/issues/447](https://github.com/docker/distribution/issues/447)"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/116750450","html_url":"https://github.com/docker/distribution/issues/459#issuecomment-116750450","issue_url":"https://api.github.com/repos/docker/distribution/issues/459","id":116750450,"user":{"login":"ahmetalpbalkan","id":159209,"avatar_url":"https://avatars.githubusercontent.com/u/159209?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmetalpbalkan","html_url":"https://github.com/ahmetalpbalkan","followers_url":"https://api.github.com/users/ahmetalpbalkan/followers","following_url":"https://api.github.com/users/ahmetalpbalkan/following{/other_user}","gists_url":"https://api.github.com/users/ahmetalpbalkan/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetalpbalkan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetalpbalkan/subscriptions","organizations_url":"https://api.github.com/users/ahmetalpbalkan/orgs","repos_url":"https://api.github.com/users/ahmetalpbalkan/repos","events_url":"https://api.github.com/users/ahmetalpbalkan/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetalpbalkan/received_events","type":"User","site_admin":false},"created_at":"2015-06-29T16:28:14Z","updated_at":"2015-06-29T16:28:14Z","body":"@bobrik thanks for supporting my suggestion. Here's a bit more context on what we will do (as a cloud provider, Azure):\r\n\r\nAssuming the \"download and stream at the same time in case of cache miss\" is implemented, we will have bots to pull most popular N images from Docker Hub every hour or so, to fill up the cache eagerly with the missing layers and reduce cache misses statistically, in each datacenter as you said.\r\n\r\n+1 on enumerating goals and steps, timelines to prevent this proposal from stalling."}}