{"action":"created","issue":{"url":"https://api.github.com/repos/rancherio/rancher/issues/1015","labels_url":"https://api.github.com/repos/rancherio/rancher/issues/1015/labels{/name}","comments_url":"https://api.github.com/repos/rancherio/rancher/issues/1015/comments","events_url":"https://api.github.com/repos/rancherio/rancher/issues/1015/events","html_url":"https://github.com/rancherio/rancher/issues/1015","id":84151607,"number":1015,"title":"Deleted hosts are still assigned to monitor health checks for instances , resulting incorrect health check report.","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancherio/rancher/labels/area/service","name":"area/service","color":"FBCA04"},{"url":"https://api.github.com/repos/rancherio/rancher/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/rancherio/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66"}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancherio/rancher/milestones/25","html_url":"https://github.com/rancherio/rancher/milestones/Release%201.0%20Beta","labels_url":"https://api.github.com/repos/rancherio/rancher/milestones/25/labels","id":1134764,"number":25,"title":"Release 1.0 Beta","description":"","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":44,"closed_issues":144,"state":"open","created_at":"2015-05-26T21:14:50Z","updated_at":"2015-06-11T23:25:41Z","due_on":"2015-06-13T07:00:00Z","closed_at":null},"comments":1,"created_at":"2015-06-02T18:48:08Z","updated_at":"2015-06-11T23:25:41Z","closed_at":"2015-06-11T23:25:41Z","body":"Steps to reproduce the problem:\r\nSet up has 3 hosts:\r\n\r\nCreate a service with scale 2 and health check enabled.\r\nIn my case , I also use the following labels for containers within the service to be hosted in different hosts - io.rancher.scheduler.affinity:container_label_ne = io.rancher.service.name=\r\n\r\nActivated the service.\r\n\r\nAll 3 hosts were allocated to monitor the instances of this service.\r\n\r\nDelete one of the hosts.\r\n\r\nThe deleted hosts continue to be allocated for monitoring health checks of instances.\r\n\r\nWhen one of existing host was stopped , other hosts in the setup where able to report the state of system as unhealthy. But the deleted host's monitoring status for the container remains in \"healthy\" state which will prevent  any service replication from happening.\r\n\r\n\r\nmysql> select health_state , host_id from healthcheck_instance_host_map where healthcheck_instance_id=(select id from healthcheck_instance where instance_id=14);\r\n+--------------+---------+\r\n| health_state | host_id |\r\n+--------------+---------+\r\n| unhealthy    |       2 |\r\n| healthy      |       1 |\r\n| healthy      |       3 |\r\n+--------------+---------+\r\n3 rows in set (0.00 sec)\r\n\r\nmysql> select id, name, state from host;\r\n+----+------------------------+--------+\r\n| id | name                   | state  |\r\n+----+------------------------+--------+\r\n|  1 | sangee0230rc1-10acre-1 | active |\r\n|  2 | sangee0230rc1-10acre-2 | active |\r\n|  3 | sangee0230rc1-10acre-3 | purged |\r\n|  4 | DO23                   | active |\r\n+----+------------------------+--------+\r\n4 rows in set (0.00 sec)\r\n"},"comment":{"url":"https://api.github.com/repos/rancherio/rancher/issues/comments/111305450","html_url":"https://github.com/rancherio/rancher/issues/1015#issuecomment-111305450","issue_url":"https://api.github.com/repos/rancherio/rancher/issues/1015","id":111305450,"user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"created_at":"2015-06-11T23:25:41Z","updated_at":"2015-06-11T23:25:41Z","body":"Tested with server version - V0.24.0-rc1\r\n\r\nWhen host gets deleted , I see the state being marked \"removed\" in healthcheck_instance_host_map.\r\nInstances that are reported \"unhealthy\" by other 2 hosts are considered unhealthy and new instance is spawned for this.\r\n "}}