{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/1877","labels_url":"https://api.github.com/repos/rancher/rancher/issues/1877/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/1877/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/1877/events","html_url":"https://github.com/rancher/rancher/issues/1877","id":103236704,"number":1877,"title":"Dealing with Host failures","user":{"login":"Rucknar","id":5539660,"avatar_url":"https://avatars.githubusercontent.com/u/5539660?v=3","gravatar_id":"","url":"https://api.github.com/users/Rucknar","html_url":"https://github.com/Rucknar","followers_url":"https://api.github.com/users/Rucknar/followers","following_url":"https://api.github.com/users/Rucknar/following{/other_user}","gists_url":"https://api.github.com/users/Rucknar/gists{/gist_id}","starred_url":"https://api.github.com/users/Rucknar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rucknar/subscriptions","organizations_url":"https://api.github.com/users/Rucknar/orgs","repos_url":"https://api.github.com/users/Rucknar/repos","events_url":"https://api.github.com/users/Rucknar/events{/privacy}","received_events_url":"https://api.github.com/users/Rucknar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-26T10:51:24Z","updated_at":"2015-08-27T08:57:38Z","closed_at":null,"body":"I'm currently exploring non functional testing on a POC environment we have stood up using rancher.\r\nI've done a simple test to force power off one of the hosts rancher is managing, this however has highlighted a few issues. I've had a search on the issues list but couldn't find much covering them.\r\n\r\n**Expected Outcome:**\r\nI expected the host to be marked offline and i expected the services to maintain their scale by spinning up containers on other available hosts.\r\n\r\n**Actual Outcome:**\r\nRancher detected the failure very quickly and marked the host in red with a message of 'reconnecting'.\r\nHowever:\r\n- Services didn't attempt to maintain scale across other hosts, no new containers were spun up.\r\n- Applications view and infrastructure view still view the containers with a hollow green circle suggesting they are running fine.\r\n\r\n**Summary**\r\nFrom a user experience, I'd expect a host failure to be somewhat more highlighted within the GUI and i'd expect the containers which no longer exist to be highlighted in some fashion so i can quickly understand the impact of the failure. As a sidepoint and perhaps other ticket it would be nice to have a simple status traffic lights for services running through the load balancers. \r\n\r\nFrom a service recovery point of view, i'd like to be able to configure the behaviour of rancher-server here in how it manages failure scenarios. A fairly safe non aggressive default should be fine and then allow the user to customise it to their requirements.  It's an area of functionality that needs careful considerations of the logic, that said it's also something that's critical as we are assessing rancher against tools such as vSphere / MS VMM. "},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/135348759","html_url":"https://github.com/rancher/rancher/issues/1877#issuecomment-135348759","issue_url":"https://api.github.com/repos/rancher/rancher/issues/1877","id":135348759,"user":{"login":"Rucknar","id":5539660,"avatar_url":"https://avatars.githubusercontent.com/u/5539660?v=3","gravatar_id":"","url":"https://api.github.com/users/Rucknar","html_url":"https://github.com/Rucknar","followers_url":"https://api.github.com/users/Rucknar/followers","following_url":"https://api.github.com/users/Rucknar/following{/other_user}","gists_url":"https://api.github.com/users/Rucknar/gists{/gist_id}","starred_url":"https://api.github.com/users/Rucknar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rucknar/subscriptions","organizations_url":"https://api.github.com/users/Rucknar/orgs","repos_url":"https://api.github.com/users/Rucknar/repos","events_url":"https://api.github.com/users/Rucknar/events{/privacy}","received_events_url":"https://api.github.com/users/Rucknar/received_events","type":"User","site_admin":false},"created_at":"2015-08-27T08:57:38Z","updated_at":"2015-08-27T08:57:38Z","body":"I've been told that we should actually be seeing containers restart on other hosts, i'm wondering if there is something wrong with our configuration. All of our containers have health checks on the container and at the loadbalancer level. Example below, some config \r\n\r\n```\r\nAudit:\r\n  restart: always\r\n  labels:\r\n    io.rancher.container.hostname_override: container_name\r\n  tty: true\r\n  image: registry:5000/aud:65\r\n  stdin_open: true\r\n  volumes:\r\n    - /home/rancher/env_config/rt/tomcat/catalina.sh:/usr/share/secure_tomcat/bin/catalina.sh\r\n    - /home/rancher/gluster:/mnt/gluster\r\n    - /home/rancher/env_config/rt:/home/***_user/***\r\n  links:\r\n    - IAM-Database:***\r\n    - LB-***:LB-***\r\n    - LB-***:LB-***\r\n```"}}