{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/12198","labels_url":"https://api.github.com/repos/docker/docker/issues/12198/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/12198/comments","events_url":"https://api.github.com/repos/docker/docker/issues/12198/events","html_url":"https://github.com/docker/docker/pull/12198","id":67226271,"number":12198,"title":"Add MARK and SQUASH builder instructions","user":{"login":"burke","id":1284,"avatar_url":"https://avatars.githubusercontent.com/u/1284?v=3","gravatar_id":"","url":"https://api.github.com/users/burke","html_url":"https://github.com/burke","followers_url":"https://api.github.com/users/burke/followers","following_url":"https://api.github.com/users/burke/following{/other_user}","gists_url":"https://api.github.com/users/burke/gists{/gist_id}","starred_url":"https://api.github.com/users/burke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burke/subscriptions","organizations_url":"https://api.github.com/users/burke/orgs","repos_url":"https://api.github.com/users/burke/repos","events_url":"https://api.github.com/users/burke/events{/privacy}","received_events_url":"https://api.github.com/users/burke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/status/1-design-review","name":"status/1-design-review","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":58,"created_at":"2015-04-08T21:42:15Z","updated_at":"2015-12-08T15:52:34Z","closed_at":"2015-05-14T18:51:21Z","pull_request":{"url":"https://api.github.com/repos/docker/docker/pulls/12198","html_url":"https://github.com/docker/docker/pull/12198","diff_url":"https://github.com/docker/docker/pull/12198.diff","patch_url":"https://github.com/docker/docker/pull/12198.patch"},"body":"## TL;DR\r\n\r\nThere are actually docs in the diff, but the 10-second overview is:\r\n\r\n```Dockerfile\r\nFROM scratch\r\nMAINTAINER me\r\nMARK\r\nADD a\r\nRUN b\r\nSQUASH \">> important things <<\"\r\n```\r\n\r\n```\r\n$ docker history ${id}\r\nIMAGE               CREATED                  CREATED BY                          SIZE\r\n2a1491d233a1        Less than a second ago   >> important things <<              20 KB\r\ndf7546f9f060        6 months ago             /bin/sh -c #(nop) MAINTAINER me     0 B\r\n511136ea3c5a        22 months ago                                                0 B\r\n$\r\n```\r\n\r\n## Summary\r\n\r\nThis PR adds two commands -- `MARK` and `SQUASH` -- to the builder. I've wanted some way to implement \"transactions\", whereby multiple instructions are ultimately reduced to a single layer, in `Dockerfile`s for a long time. This was by far the least invasive way to implement it.\r\n\r\n* `MARK` stashes a tiny bit of context on the `Builder` struct so that `SQUASH` can look it up later.\r\n* `SQUASH` squashes the indicated layers between `MARK` and `SQUASH` together into a single new layer. \r\n* `SQUASH` also gives the new layer a new `ContainerConfig.Cmd`, whose only apparent use is to show up in `docker history`. This can be used to substantially prettify `docker history` output.\r\n\r\n## Why is this necessary?\r\n\r\n1. At Shopify, some of our containers require secrets to be temporarily decrypted during build time. There are trivially-conceivable three strategies for this. Two of them are impossible currently, and one is sketchy. See Appendix A. `SQUASH` makes it possible to have temporary build-time secrets without them existing in the image hierarchy.\r\n\r\n2. It's fairly common to want to `ADD` some files, take some action with them, then to remove them. This causes the final image distribution to be much larger than necessary. If this can be compressed to a single layer, network and disk are used more efficiently.\r\n\r\n3. At Shopify, we enjoy having prettified image histories. For example, this is our current core platform production image history:\r\n\r\n```\r\nIMAGE               CREATED             CREATED BY                                      SIZE\r\n3a342817ba4a        37 minutes ago      ##build 3 & finalize @d9c88f1c                  x MB\r\n4a4f06aab8d0        9 hours ago         ##build 2 @1d9a653a                             y MB\r\n5e004daccb40        2 days ago          ##build 1 @a1ecb04a                             z MB\r\n38fee11f123e        2 weeks ago         ##prepare                                       w MB\r\n623b6b3205b7        2 weeks ago         ##baseimage v1.3.11                             v MB\r\nd0955f21bf24        2 weeks ago         /bin/sh -c #(nop) CMD [/bin/bash]               0 B\r\n9fec74352904        2 weeks ago         /bin/sh -c sed -i 's/^#\\s*\\(deb.*universe\\)$/   1.895 kB\r\na1a958a24818        2 weeks ago         /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB\r\nf3c84ac3a053        2 weeks ago         /bin/sh -c #(nop) ADD file:777fad733fc954c0c1   188.1 MB\r\n511136ea3c5a        22 months ago                                                       0 B\r\n```\r\n\r\nThis `MARK`/`SQUASH` would make our rather insane custom build process theoretically-achievable using only a handful of strategically-composed `Dockerfile`s.\r\n\r\n## Conclusions...\r\n\r\nThere's still a lot of testing and polish to be applied here, but before I continue, I want to get a sense of how likely this is to be accepted; i.e. is the docker project still philosophically opposed to acknowledging the existence of layers from the context of a `Dockerfile`?\r\n\r\nDoes this seem like the kind of that that could possibly move forward?\r\n\r\n/cc Shopify peeps: @graemej @Sirupsen @thegedge @shivnagarajan\r\n\r\n---\r\n\r\n#### Appendix A: Obvious ways to get secrets during a build\r\n\r\n1. Volumes. Volumes don't work during build.\r\n\r\n2. `ADD` them, `RUN` something, then remove the secrets. This has the unfortunate effect of the secrets existing in plaintext in the `ADD` layer. This is the case that `SQUASH` ameliorates.\r\n\r\n3. Operate a HTTP server, poke the secrets into it, and have the build retrieve, use, and delete them in a single `RUN` command. We did this in the past, and I've heard of others doing it, but I find it rather scary."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/162924865","html_url":"https://github.com/docker/docker/pull/12198#issuecomment-162924865","issue_url":"https://api.github.com/repos/docker/docker/issues/12198","id":162924865,"user":{"login":"duglin","id":1944671,"avatar_url":"https://avatars.githubusercontent.com/u/1944671?v=3","gravatar_id":"","url":"https://api.github.com/users/duglin","html_url":"https://github.com/duglin","followers_url":"https://api.github.com/users/duglin/followers","following_url":"https://api.github.com/users/duglin/following{/other_user}","gists_url":"https://api.github.com/users/duglin/gists{/gist_id}","starred_url":"https://api.github.com/users/duglin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duglin/subscriptions","organizations_url":"https://api.github.com/users/duglin/orgs","repos_url":"https://api.github.com/users/duglin/repos","events_url":"https://api.github.com/users/duglin/events{/privacy}","received_events_url":"https://api.github.com/users/duglin/received_events","type":"User","site_admin":false},"created_at":"2015-12-08T15:52:33Z","updated_at":"2015-12-08T15:52:33Z","body":"@cpuguy83 you've mention this on IRC too and I keep meaning to ask you about this. How are the layers gone? When I build an image I still the layers in `docker history` and I still see the \"parent\" field in docker inspect` for the images/layers.  What appears to be different is that they're now sha's instead of just random IDs.  When I walk the history to an \"old\" image (like ubuntu) things get a bit weird, but I assumed this was because of the migration stuff, but for \"new\" images it sure looks like the old stuff just with new types of IDs.  Can you elaborate on how things are different?"}}