{"action":"closed","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/2637","labels_url":"https://api.github.com/repos/rancher/rancher/issues/2637/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/2637/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/2637/events","html_url":"https://github.com/rancher/rancher/issues/2637","id":116402616,"number":2637,"title":"Healthy instances get marked as \"unhealthy\" and recreated.","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"},{"url":"https://api.github.com/repos/rancher/rancher/labels/status/reopened","name":"status/reopened","color":"C80408"},{"url":"https://api.github.com/repos/rancher/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66"}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/47","html_url":"https://github.com/rancher/rancher/milestones/Milestone%2011/18/2015","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/47/labels","id":1392249,"number":47,"title":"Milestone 11/18/2015","description":"0.47","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":5,"state":"open","created_at":"2015-11-03T21:14:04Z","updated_at":"2015-11-13T23:40:13Z","due_on":"2015-11-18T08:00:00Z","closed_at":null},"comments":6,"created_at":"2015-11-11T19:18:44Z","updated_at":"2015-11-13T23:40:13Z","closed_at":"2015-11-13T23:40:13Z","body":"Server version - Build from master\r\n\r\nSteps to reproduce the problem:\r\n\r\nSet up has 2 hosts.\r\nHad a service with scale 15.\r\nThere were few instances in host1 and few in host2.\r\n\r\nStopped host1.\r\n\r\nThis resulted in existing instances in host1 being marked for \"deletion\" and new instances being created in host2 , so that all instances of the service are in host2.\r\n\r\nStarted host1.\r\n\r\nWhen host1 got to \"Active\" state , I see the instances that are stuck in \"removing\" state getting removed. \r\nBut this also triggered few existing instances in host2 to be marked \"unhealthy\" , which triggered new instances to be created in host2.\r\n\r\n\r\n\r\n```\r\nAfter host1 went to \"reconnecting\" state and healthcheck triggered new instances to be created.\r\n\r\nmysql> select id,name,state,health_state from instance where name like \"%myhealth%\";\r\n+-----+------------------------+---------+--------------+\r\n| id  | name                   | state   | health_state |\r\n+-----+------------------------+---------+--------------+\r\n| 224 | test494388_myhealth_1  | purged  | unhealthy    |\r\n| 225 | test494388_myhealth_2  | purged  | unhealthy    |\r\n| 226 | test494388_myhealth_3  | purged  | unhealthy    |\r\n| 227 | test494388_myhealth_4  | purged  | unhealthy    |\r\n| 228 | test494388_myhealth_5  | purging | unhealthy    |\r\n| 229 | test494388_myhealth_6  | purged  | healthy      |\r\n| 230 | test494388_myhealth_7  | purging | unhealthy    |\r\n| 231 | test494388_myhealth_1  | purging | unhealthy    |\r\n| 232 | test494388_myhealth_2  | purging | unhealthy    |\r\n| 237 | test494388_myhealth_3  | running | healthy      |\r\n| 238 | test494388_myhealth_4  | running | healthy      |\r\n| 239 | test494388_myhealth_6  | running | healthy      |\r\n| 240 | test494388_myhealth_8  | running | healthy      |\r\n| 241 | test494388_myhealth_9  | purging | unhealthy    |\r\n| 242 | test494388_myhealth_10 | running | healthy      |\r\n| 243 | test494388_myhealth_11 | running | healthy      |\r\n| 244 | test494388_myhealth_12 | purging | unhealthy    |\r\n| 245 | test494388_myhealth_13 | purging | unhealthy    |\r\n| 246 | test494388_myhealth_14 | running | healthy      |\r\n| 247 | test494388_myhealth_15 | purging | unhealthy    |\r\n| 248 | test494388_myhealth_16 | running | initializing |\r\n| 249 | test494388_myhealth_17 | purging | initializing |\r\n| 250 | test494388_myhealth_17 | running | healthy      |\r\n| 251 | test494388_myhealth_18 | running | healthy      |\r\n| 252 | test494388_myhealth_19 | running | healthy      |\r\n| 253 | test494388_myhealth_20 | running | healthy      |\r\n| 254 | test494388_myhealth_21 | running | healthy      |\r\n| 255 | test494388_myhealth_22 | running | healthy      |\r\n| 256 | test494388_myhealth_23 | running | healthy      |\r\n+-----+------------------------+---------+--------------+\r\n29 rows in set (0.00 sec)\r\n\r\nAfter host1 went to \"Active\" state. I see 7 instances being recreated in host1.\r\n\r\nmysql> select id,name,state,health_state from instance where name like \"%myhealth%\";\r\n\r\n+-----+------------------------+---------+--------------+\r\n\r\n| id  | name                   | state   | health_state |\r\n\r\n+-----+------------------------+---------+--------------+\r\n\r\n| 224 | test494388_myhealth_1  | purged  | unhealthy    |\r\n\r\n| 225 | test494388_myhealth_2  | purged  | unhealthy    |\r\n\r\n| 226 | test494388_myhealth_3  | purged  | unhealthy    |\r\n\r\n| 227 | test494388_myhealth_4  | purged  | unhealthy    |\r\n\r\n| 228 | test494388_myhealth_5  | purged  | unhealthy    |\r\n\r\n| 229 | test494388_myhealth_6  | purged  | healthy      |\r\n\r\n| 230 | test494388_myhealth_7  | purged  | unhealthy    |\r\n\r\n| 231 | test494388_myhealth_1  | purged  | unhealthy    |\r\n\r\n| 232 | test494388_myhealth_2  | purged  | unhealthy    |\r\n\r\n| 237 | test494388_myhealth_3  | removed | unhealthy    |\r\n\r\n| 238 | test494388_myhealth_4  | removed | unhealthy    |\r\n\r\n| 239 | test494388_myhealth_6  | removed | unhealthy    |\r\n\r\n| 240 | test494388_myhealth_8  | removed | unhealthy    |\r\n\r\n| 241 | test494388_myhealth_9  | purged  | unhealthy    |\r\n\r\n| 242 | test494388_myhealth_10 | removed | unhealthy    |\r\n\r\n| 243 | test494388_myhealth_11 | removed | unhealthy    |\r\n\r\n| 244 | test494388_myhealth_12 | purged  | unhealthy    |\r\n\r\n| 245 | test494388_myhealth_13 | purged  | unhealthy    |\r\n\r\n| 246 | test494388_myhealth_14 | removed | unhealthy    |\r\n\r\n| 247 | test494388_myhealth_15 | purged  | unhealthy    |\r\n\r\n| 248 | test494388_myhealth_16 | running | healthy      |\r\n\r\n| 249 | test494388_myhealth_17 | purged  | initializing |\r\n\r\n| 250 | test494388_myhealth_17 | running | healthy      |\r\n\r\n| 251 | test494388_myhealth_18 | running | healthy      |\r\n\r\n| 252 | test494388_myhealth_19 | running | healthy      |\r\n\r\n| 253 | test494388_myhealth_20 | running | healthy      |\r\n\r\n| 254 | test494388_myhealth_21 | running | healthy      |\r\n\r\n| 255 | test494388_myhealth_22 | running | healthy      |\r\n\r\n| 256 | test494388_myhealth_23 | running | healthy      |\r\n\r\n| 265 | test494388_myhealth_1  | running | healthy      |\r\n\r\n| 266 | test494388_myhealth_2  | running | healthy      |\r\n\r\n| 267 | test494388_myhealth_3  | running | healthy      |\r\n\r\n| 268 | test494388_myhealth_4  | running | healthy      |\r\n\r\n| 269 | test494388_myhealth_5  | running | healthy      |\r\n\r\n| 270 | test494388_myhealth_7  | running | healthy      |\r\n\r\n| 271 | test494388_myhealth_9  | running | healthy      |\r\n\r\n+-----+------------------------+---------+--------------+\r\n```\r\n"}}