{"push_id":668865062,"size":11,"distinct_size":1,"ref":"refs/heads/MoveErrors","head":"3b398da729c4a6a6b7e478dcb357d869ccc84c92","before":"cb47cc48fff6a87456bbc78858e070b8fb86c862","commits":[{"sha":"593bbccdb5820a6afb1e8e17931a5210ebf0cf6a","author":{"email":"63efd86fa22882ae89b6027b22a36e4db1f4214d@docker.com","name":"Stephen J Day"},"message":"Refactor Blob Service API\n\nThis PR refactors the blob service API to be oriented around blob descriptors.\nIdentified by digests, blobs become an abstract entity that can be read and\nwritten using a descriptor as a handle. This allows blobs to take many forms,\nsuch as a ReadSeekCloser or a simple byte buffer, allowing blob oriented\noperations to better integrate with blob agnostic APIs (such as the `io`\npackage). The error definitions are now better organized to reflect conditions\nthat can only be seen when interacting with the blob API.\n\nThe main benefit of this is to separate the much smaller metadata from large\nfile storage. Many benefits also follow from this. Reading and writing has\nbeen separated into discrete services. Backend implementation is also\nsimplified, by reducing the amount of metadata that needs to be picked up to\nsimply serve a read. This also improves cacheability.\n\n\"Opening\" a blob simply consists of an access check (Stat) and a path\ncalculation. Caching is greatly simplified and we've made the mapping of\nprovisional to canonical hashes a first-class concept. BlobDescriptorService\nand BlobProvider can be combined in different ways to achieve varying effects.\n\nRecommend Review Approach\n-------------------------\n\nThis is a very large patch. While apologies are in order, we are getting a\nconsiderable amount of refactoring. Most changes follow from the changes to\nthe root package (distribution), so start there. From there, the main changes\nare in storage. Looking at (*repository).Blobs will help to understand the how\nthe linkedBlobStore is wired. One can explore the internals within and also\nbranch out into understanding the changes to the caching layer. Following the\ndescriptions below will also help to guide you.\n\nTo reduce the chances for regressions, it was critical that major changes to\nunit tests were avoided. Where possible, they are left untouched and where\nnot, the spirit is hopefully captured. Pay particular attention to where\nbehavior may have changed.\n\nStorage\n-------\n\nThe primary changes to the `storage` package, other than the interface\nupdates, were to merge the layerstore and blobstore. Blob access is now\nlayered even further. The first layer, blobStore, exposes a global\n`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most\nread operations that don't take access control into account. The\n`linkedBlobStore` layers on top of the `blobStore`, providing repository-\nscoped blob link management in the backend. The `linkedBlobStore` implements\nthe full `BlobStore` suite, providing access-controlled, repository-local blob\nwriters. The abstraction between the two is slightly broken in that\n`linkedBlobStore` is the only channel under which one can write into the global\nblob store. The `linkedBlobStore` also provides flexibility in that it can act\nover different link sets depending on configuration. This allows us to use the\nsame code for signature links, manifest links and blob links.  Eventually, we\nwill fully consolidate this storage.\n\nThe improved cache flow comes from the `linkedBlobStatter` component\nof `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to\nprovide a simple cache hierarchy that should streamline access checks on read\nand write operations, or at least provide a single path to optimize. The\nmetrics have been changed in a slightly incompatible way since the former\noperations, Fetch and Exists, are no longer relevant.\n\nThe fileWriter and fileReader have been slightly modified to support the rest\nof the changes. The most interesting is the removal of the `Stat` call from\n`newFileReader`. This was the source of unnecessary round trips that were only\npresent to look up the size of the resulting reader. Now, one must simply pass\nin the size, requiring the caller to decide whether or not the `Stat` call is\nappropriate. In several cases, it turned out the caller already had the size\nalready. The `WriterAt` implementation has been removed from `fileWriter`,\nsince it is no longer required for `BlobWriter`, reducing the number of paths\nwhich writes may take.\n\nCache\n-----\n\nUnfortunately, the `cache` package required a near full rewrite. It was pretty\nmechanical in that the cache is oriented around the `BlobDescriptorService`\nslightly modified to include the ability to set the values for individual\ndigests. While the implementation is oriented towards caching, it can act as a\nprimary store. Provisions are in place to have repository local metadata, in\naddition to global metadata. Fallback is implemented as a part of the storage\npackage to maintain this flexibility.\n\nOne unfortunate side-effect is that caching is now repository-scoped, rather\nthan global. This should have little effect on performance but may increase\nmemory usage.\n\nHandlers\n--------\n\nThe `handlers` package has been updated to leverage the new API. For the most\npart, the changes are superficial or mechanical based on the API changes. This\ndid expose a bug in the handling of provisional vs canonical digests that was\nfixed in the unit tests.\n\nConfiguration\n-------------\n\nOne user-facing change has been made to the configuration and is updated in\nthe associated documentation. The `layerinfo` cache parameter has been\ndeprecated by the `blobdescriptor` cache parameter. Both are equivalent and\nconfiguration files should be backward compatible.\n\nNotifications\n-------------\n\nChanges the `notification` package are simply to support the interface\nchanges.\n\nContext\n-------\n\nA small change has been made to the tracing log-level. Traces have been moved\nfrom \"info\" to \"debug\" level to reduce output when not needed.\n\nSigned-off-by: Stephen J Day <stephen.day@docker.com>","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/593bbccdb5820a6afb1e8e17931a5210ebf0cf6a"},{"sha":"ace71ea9c99b957332a525d60366c380f544d5fd","author":{"email":"e410617591fe95118a52463efecfc8bda1cabae9@users.noreply.github.com","name":"Stephen Day"},"message":"Merge pull request #519 from stevvooe/blob-service-refactor\n\nRefactor Blob Service API","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/ace71ea9c99b957332a525d60366c380f544d5fd"},{"sha":"beab2d7fbebc1ebb84d1ba0bfd7dfef0c4fd2e40","author":{"email":"e1c79a582b6629e6b39e9679f4bb964d25db4aa8@mcgstyle.net","name":"Derek McGowan"},"message":"Remove set header in compose script\n\nSigned-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/beab2d7fbebc1ebb84d1ba0bfd7dfef0c4fd2e40"},{"sha":"8d407c81aeef8d5f5a40b2e7e17ee0c511d03bb6","author":{"email":"5665331b9b819ac358165f8c38970dc8c7ddb47d@docker.com","name":"Mary Anthony"},"message":"Closes #485\n\nSigned-off-by: Mary Anthony <mary@docker.com>\n\nTweak per Stephen\n\nSigned-off-by: Mary Anthony <mary@docker.com>","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/8d407c81aeef8d5f5a40b2e7e17ee0c511d03bb6"},{"sha":"4d17981206f7561f63c8b32e202ee6bf77b53021","author":{"email":"e410617591fe95118a52463efecfc8bda1cabae9@users.noreply.github.com","name":"Stephen Day"},"message":"Merge pull request #529 from moxiegirl/485-issue-fix\n\ndocs: add warning about using window's volumes with registry","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/4d17981206f7561f63c8b32e202ee6bf77b53021"},{"sha":"316f3c58316fd7efa59c7d45bd2aad77dc095e23","author":{"email":"e410617591fe95118a52463efecfc8bda1cabae9@users.noreply.github.com","name":"Stephen Day"},"message":"Merge pull request #530 from dmcgowan/fix-compose-nginx\n\nRemove set header in compose script","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/316f3c58316fd7efa59c7d45bd2aad77dc095e23"},{"sha":"e1a1e3a8dacc70b1f745663a31fab696d2f2b28a","author":{"email":"5665331b9b819ac358165f8c38970dc8c7ddb47d@docker.com","name":"Mary Anthony"},"message":"Fixes #484\n\nSigned-off-by: Mary Anthony <mary@docker.com>","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/e1a1e3a8dacc70b1f745663a31fab696d2f2b28a"},{"sha":"cde7b93b8560b30966171e0135e5be05a87d1e39","author":{"email":"e410617591fe95118a52463efecfc8bda1cabae9@users.noreply.github.com","name":"Stephen Day"},"message":"Merge pull request #531 from moxiegirl/484-version-fix\n\ndocs: clarify minimum docker version for use with v2 API","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/cde7b93b8560b30966171e0135e5be05a87d1e39"},{"sha":"32ed2d7df1a88c06865d628b6074fd85c9fa2575","author":{"email":"aa6ef2bbe17379c3e8e7883829f55c5a7ad052b1@redhat.com","name":"Jordan Liggitt"},"message":"Fix typo: respository->repository\n\nSigned-off-by: Jordan Liggitt <jliggitt@redhat.com>","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/32ed2d7df1a88c06865d628b6074fd85c9fa2575"},{"sha":"36d4e365aabb7197687a84a8a24127c71eefd639","author":{"email":"e410617591fe95118a52463efecfc8bda1cabae9@users.noreply.github.com","name":"Stephen Day"},"message":"Merge pull request #534 from liggitt/respository_typo\n\nFix typo: respository->repository","distinct":false,"url":"https://api.github.com/repos/duglin/distribution/commits/36d4e365aabb7197687a84a8a24127c71eefd639"},{"sha":"3b398da729c4a6a6b7e478dcb357d869ccc84c92","author":{"email":"ff3bde741799c0eff815b83a54eda3fc494df54f@us.ibm.com","name":"Doug Davis"},"message":"Move ErrorCode logic to new errcode package\n\nSigned-off-by: Doug Davis <dug@us.ibm.com>","distinct":true,"url":"https://api.github.com/repos/duglin/distribution/commits/3b398da729c4a6a6b7e478dcb357d869ccc84c92"}]}