{"action":"opened","issue":{"url":"https://api.github.com/repos/GELOG/adamcloud/issues/18","labels_url":"https://api.github.com/repos/GELOG/adamcloud/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/GELOG/adamcloud/issues/18/comments","events_url":"https://api.github.com/repos/GELOG/adamcloud/issues/18/events","html_url":"https://github.com/GELOG/adamcloud/issues/18","id":69684316,"number":18,"title":"Docker image repeatability","user":{"login":"davidonlaptop","id":10860,"avatar_url":"https://avatars.githubusercontent.com/u/10860?v=3","gravatar_id":"","url":"https://api.github.com/users/davidonlaptop","html_url":"https://github.com/davidonlaptop","followers_url":"https://api.github.com/users/davidonlaptop/followers","following_url":"https://api.github.com/users/davidonlaptop/following{/other_user}","gists_url":"https://api.github.com/users/davidonlaptop/gists{/gist_id}","starred_url":"https://api.github.com/users/davidonlaptop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidonlaptop/subscriptions","organizations_url":"https://api.github.com/users/davidonlaptop/orgs","repos_url":"https://api.github.com/users/davidonlaptop/repos","events_url":"https://api.github.com/users/davidonlaptop/events{/privacy}","received_events_url":"https://api.github.com/users/davidonlaptop/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GELOG/adamcloud/labels/discussion","name":"discussion","color":"e11d21"},{"url":"https://api.github.com/repos/GELOG/adamcloud/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-20T21:23:50Z","updated_at":"2015-04-20T21:23:50Z","closed_at":null,"body":"# Current setup\r\nThis diagram shows the inheritance between Docker images, and which images may use another image.\r\n![image hierarchy](http://yuml.me/diagram/plain;dir:LR/class/[SNAP:1.0beta15]-^[Ubuntu:14.04.1], [ADAM:0.14]-^[Spark:1.1-bin-hadoop2.3], [ADAM:0.15]-^[Spark:1.2-bin-hadoop2.3], [Avocado:0.0-master-branch-s1.1-h2.3]-^[Spark:1.1-bin-hadoop2.3], [Avocado:0.0-master-branch-s1.2-h2.3]-^[Spark:1.2-bin-hadoop2.3], [Spark:1.1-bin-hadoop2.3]-^[Java:openjdk7], [Spark:1.1-bin-hadoop2.4]-^[Java:openjdk7], [Spark:1.2-bin-hadoop2.3]-^[Java:openjdk7], [Java:openjdk7]-^[Ubuntu:14.04.1], [Hadoop:2.3]-^[Java:openjdk7], [Hadoop:2.6]-^[Java:openjdk7], [Spark:1.1-bin-hadoop2.3]->[Hadoop:2.3], [Spark:1.2-bin-hadoop2.3]->[Hadoop:2.3])\r\n\r\n# Repeatability with Docker image\r\n\r\n## The issue\r\nSome images are building from source (e.g. avocado builds from GitHub sources). This means that when we want to update the Dockerfile for an already published image on the Docker Hub, the resulting code may have changed beyond our control. This can cause unexpected behavior to the users of the image, and can perhaps even break user's pipeline.\r\n\r\nTherefore, we established a set of rules below to provide **_better_** repeatability.\r\n\r\n(_As for configuration management systems, it would be really difficult to reach 100% repeatability guarantee, but good-enough guarantee seems acceptable._)\r\n\r\n## Rules to improve repeatability\r\n\r\n1. All images MUST be **versioned using the encapsulated technology's version** (the image for Hadoop `2.5.1` would be named `hadoop:2.5.1`.\r\n2. **[Semantic Versioning](http://semver.org/) is assumed** unless otherwise specified (e.g. the version number follows the `MAJOR.MINOR.PATCH` convention, where breaking changes are only allowed in `MAJOR` increment).\r\n3. Images with **complete version number (e.g. `2.5.1`) SHOULD NEVER change** once published to the Docker Hub. End-users are encouraged to use these images for production use.\r\n4. **Only image aliases (e.g. `2.5`, `2`) are allowed to change**. At first `hadoop:2`, `hadoop:2.5`, and `hadoop:2.5.1` may all point to the same image. When the image `hadoop:2.5.2` is released, the images `hadoop:2` and `hadoop:2.5` are updated.\r\n5. **No guarantee** is made to provide a Docker image for all possible versions. Updates are made based on our needs or upon request.\r\n6. The folders in the **git repo follows the `MAJOR.MINOR` version scheme**. Therefore, the folder `hadoop-2.5/` would contain the current latest release (e.g. Hadoop `2.5.1`) available at image creation time.\r\n7. **Complete version numbers can be found via a GitHub tag* ***. e.g. Hadoop `2.5.1`'s Dockerfile can be found at `/tree/v2.5.1/hadoop-2.5/Dockerfile`.\r\n8. An image with a **complete version number MUST only inherits from another complete version**. e.g. `spark:1.3.0` can inherit from `hadoop:2.3.0` but not `hadoop:2.3` nor `hadoop:2`.\r\n9. The Dockerfile should **specify the versions of all software that it install**. If the Hadoop image requires maven to build, it should specify the maven version in a variable of the Dockerfile. As well as the Hadoop version.\r\n10. When possible, **all images should inherit from the same based image**. For now, it is Ubuntu `14.04.1`.\r\n11. When possible, **all Java-based software should use OpenJDK** as OracleJDK cannot be distributed legally in a Docker image without the user accepting the license (see [explaination from Docker team here](https://registry.hub.docker.com/_/java/)).\r\n12. A special care should taken to produce the **image size as small as possible**. Refer to the [advices for building official Docker images](https://docs.docker.com/docker-hub/official_repos/).\r\n\r\n\r\n## Remaining issues (Questions for ADAM Team)\r\n\r\n### Q1) Adam inherits from Spark, but Spark has multiple versions.\r\nThis can multiply the number of images to support. If there is 10 images of Spark, and 10 images of Adam, that makes 100 possible images to support. This will become quickly unmaintainable.\r\n\r\n**Is there a way to build a Docker image for Adam that will make it compatible with multiple Docker images of Spark ?**\r\n\r\ne.g. If `Adam 0.15.0` image inherits from `Spark 1.2.1` image, can it be used with a Spark cluster with a version less or equal to 1.2.1 ?\r\n\r\n### Q2) Spark has multiple platforms\r\nSimilar issue as above. A single release for Spark can be built for multiple platforms: Hadoop 2.3, Hadoop 2.4, Hadoop 2.6, MapR 3.x, MapR 4.x, Hadoop 1.x, CDH4, etc...\r\n\r\nSupporting MapR, CDH4, Hadoop 1.x is separate issue, out of the scope of this analysis. This leaves us with 3 releases of Hadoop.\r\n\r\n**Is there a way to build a docker image for Spark that will make it compatible with multiple Docker images of Hadoop?**\r\n\r\nOtherwise, we need to build at least 3 images for each new version of Spark."}}