{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarm/issues/1488","labels_url":"https://api.github.com/repos/docker/swarm/issues/1488/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarm/issues/1488/comments","events_url":"https://api.github.com/repos/docker/swarm/issues/1488/events","html_url":"https://github.com/docker/swarm/issues/1488","id":120116728,"number":1488,"title":"Proposal: Container Rescheduling","user":{"login":"aluzzardi","id":179503,"avatar_url":"https://avatars.githubusercontent.com/u/179503?v=3","gravatar_id":"","url":"https://api.github.com/users/aluzzardi","html_url":"https://github.com/aluzzardi","followers_url":"https://api.github.com/users/aluzzardi/followers","following_url":"https://api.github.com/users/aluzzardi/following{/other_user}","gists_url":"https://api.github.com/users/aluzzardi/gists{/gist_id}","starred_url":"https://api.github.com/users/aluzzardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aluzzardi/subscriptions","organizations_url":"https://api.github.com/users/aluzzardi/orgs","repos_url":"https://api.github.com/users/aluzzardi/repos","events_url":"https://api.github.com/users/aluzzardi/events{/privacy}","received_events_url":"https://api.github.com/users/aluzzardi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/swarm/labels/area/scheduler","name":"area/scheduler","color":"fbca04"},{"url":"https://api.github.com/repos/docker/swarm/labels/kind/proposal","name":"kind/proposal","color":"207de5"}],"state":"open","locked":false,"assignee":{"login":"vieux","id":1032519,"avatar_url":"https://avatars.githubusercontent.com/u/1032519?v=3","gravatar_id":"","url":"https://api.github.com/users/vieux","html_url":"https://github.com/vieux","followers_url":"https://api.github.com/users/vieux/followers","following_url":"https://api.github.com/users/vieux/following{/other_user}","gists_url":"https://api.github.com/users/vieux/gists{/gist_id}","starred_url":"https://api.github.com/users/vieux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vieux/subscriptions","organizations_url":"https://api.github.com/users/vieux/orgs","repos_url":"https://api.github.com/users/vieux/repos","events_url":"https://api.github.com/users/vieux/events{/privacy}","received_events_url":"https://api.github.com/users/vieux/received_events","type":"User","site_admin":false},"milestone":null,"comments":15,"created_at":"2015-12-03T08:24:50Z","updated_at":"2015-12-14T15:14:21Z","closed_at":null,"body":"## Background\r\n\r\nThe goal of this proposal is to reschedule containers automatically in case of node failure.\r\n\r\nThis is currently one of the top requested feature for Swarm.\r\n\r\n## Configuration\r\n\r\nThe behavior should be user controllable and **disabled** by default since rescheduling can have nasty effects on stateful containers.\r\n\r\nThe user can select the policy at `run` time use the `reschedule` environment variable:\r\n\r\n```\r\ndocker run -e reschedule:on-node-failure redis\r\n```\r\n\r\nPossible values for `reschedule` are:\r\n- **no**: Never reschedule the container (**default**).\r\n- **on-node-failure**: Reschedule the container whenever the node fails.\r\n\r\nThe reason this is more complicated than `yes`/`no` is in the future we might have more complicated rescheduling policies (for instance, we might want reschedule containers to re-spread or re-pack them). **Open question: Is this really necessary?**\r\n\r\nRescheduling policies will be stored as a container label: `com.docker.swarm.reschedule-policy`\r\n\r\n## Persistance\r\n\r\nIdeally, Swarm would store all containers (at least those that should be rescheduled) persistently. That way, the manager can figure out which containers are down and take action.\r\n\r\nUnfortunately, we currently don't have a shared state and this feature has been postponed because of that for a long time.\r\n\r\nSince this is one of the top requested feature, I propose we take a different approach until we have shared state (that feature has been postponed for usability concerns - we don't want to make a kv store a dependency for Swarm).\r\n\r\nBy storing the *rescheduling policy* as a container label, we are able to reconstruct the desired state at startup time.\r\n\r\nSince we are already storing **constraints**,  **affinities** etc as container labels (exactly for this reason), the manager will have all the information it needs to perform rescheduling.\r\n\r\nThis means we can restart the manager as much as we want and it will resume rescheduling as expected.\r\n\r\nHowever, the problem arises when a node goes **down** while the manager is not running: in that case, we won't \"remember\" that container even existed when the manager is started again.\r\n\r\nThis situation can be counter-balanced by using `replication`. The `rescheduler` would be running on the **primary** manager and, upon failure, the **replica** that gets elected **primary** would be taking over rescheduling responsibilities.\r\n\r\nSince every manager is aware of the cluster state (containers & rescheduling policy), it means that as long as at least one manager is still running we won't forget about containers.\r\n\r\n## Failure detection\r\n\r\nThis functionality is already provided by `cluster/engine`.\r\n\r\n`Engine` actively heartbeats nodes in the cluster every `X` seconds. After `Y` failures, the node is marked as *unhealthy*.\r\n\r\nRescheduling can rely on the health status already available.\r\n\r\n## Resurrection\r\n\r\nEventually, a node may come back to life and re-join the cluster. If the node has containers that were rescheduled, we will end up with duplicates.\r\n\r\nSwarm should monitor incoming nodes and, upon detecting a duplicate container, it should destroy the oldest one (keeping the most recently created container alive). This behavior could eventually be configurable by the user  (keep oldest, keep newer, ...), although we may want to avoid providing that option until we see a valid use case.\r\n\r\nIf duplicate containers were started with a `--restart` option, there is going to be a small window during which both containers are running at the same time. This can be a serious problem if only one instance of that container is supposed to run at one point.\r\n\r\nWe could force all containers that have rescheduling enabled to never automatically restart. In that case, whenever a node joins, Swarm could decide to either start the containers of destroy them if they are duplicates.\r\n\r\nHowever, there are many drawbacks to this approach:\r\n- Restart policies have to be handled by Swarm. This introduces high complexity since we'd have to re-implement things such as `--restart=on-failure:5` which require maintaining lots of state\r\n- If the manager is down, containers won't start automatically. This is a serious issue since it could lead to outages. Up until now, if Swarm is down the engines continue to operate normally and this change would break that contract.\r\n- Swarm might miss events, leading to containers not getting properly restarted\r\n\r\nFurthermore, It doesn't actually entirely solve the issue. If the node didn't actually *die* (e.g. it just froze for a while, there was a netsplit, networking temporarily dropped, ...) we will end up with duplicate containers running for a while anyway.\r\n\r\nGiven all the potential issues that might arise by handling the restart policy on the Swarm side and the fact that duplicate containers may end up running at the same time anyway, I suggest we do not interfere with `--restart` and document the fact that rescheduled containers may be running in parallel for a short time window.\r\n\r\n## Networking\r\n\r\nWhen rescheduling containers, Swarm must handle multi-host networking properly.\r\n\r\nThe goal is for the new container to *take over* the previous one.\r\n\r\nIn an overlay network setup, this may involve:\r\n- Making sure the *new* container takes over the IP address of the *old* container\r\n- Ensuring service discovery works properly\r\n- Cutting the old container off the network **before** starting the new one. Even though we presume the node to be down, it might still be up and running. Disconnecting the old container would alleviate side effects of duplicate containers."},"comment":{"url":"https://api.github.com/repos/docker/swarm/issues/comments/164462784","html_url":"https://github.com/docker/swarm/issues/1488#issuecomment-164462784","issue_url":"https://api.github.com/repos/docker/swarm/issues/1488","id":164462784,"user":{"login":"cpuguy83","id":799078,"avatar_url":"https://avatars.githubusercontent.com/u/799078?v=3","gravatar_id":"","url":"https://api.github.com/users/cpuguy83","html_url":"https://github.com/cpuguy83","followers_url":"https://api.github.com/users/cpuguy83/followers","following_url":"https://api.github.com/users/cpuguy83/following{/other_user}","gists_url":"https://api.github.com/users/cpuguy83/gists{/gist_id}","starred_url":"https://api.github.com/users/cpuguy83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpuguy83/subscriptions","organizations_url":"https://api.github.com/users/cpuguy83/orgs","repos_url":"https://api.github.com/users/cpuguy83/repos","events_url":"https://api.github.com/users/cpuguy83/events{/privacy}","received_events_url":"https://api.github.com/users/cpuguy83/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T15:14:21Z","updated_at":"2015-12-14T15:14:21Z","body":"@aluzzardi Is there something we can add to hint to swarm the scope of the volume (local vs global?)"}}