{"action":"created","issue":{"url":"https://api.github.com/repos/kelseyhightower/confd/issues/247","labels_url":"https://api.github.com/repos/kelseyhightower/confd/issues/247/labels{/name}","comments_url":"https://api.github.com/repos/kelseyhightower/confd/issues/247/comments","events_url":"https://api.github.com/repos/kelseyhightower/confd/issues/247/events","html_url":"https://github.com/kelseyhightower/confd/issues/247","id":65669417,"number":247,"title":"confd doesn't try to reconnect to etcd cluster","user":{"login":"zytek","id":3696278,"avatar_url":"https://avatars.githubusercontent.com/u/3696278?v=3","gravatar_id":"","url":"https://api.github.com/users/zytek","html_url":"https://github.com/zytek","followers_url":"https://api.github.com/users/zytek/followers","following_url":"https://api.github.com/users/zytek/following{/other_user}","gists_url":"https://api.github.com/users/zytek/gists{/gist_id}","starred_url":"https://api.github.com/users/zytek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zytek/subscriptions","organizations_url":"https://api.github.com/users/zytek/orgs","repos_url":"https://api.github.com/users/zytek/repos","events_url":"https://api.github.com/users/zytek/events{/privacy}","received_events_url":"https://api.github.com/users/zytek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kelseyhightower/confd/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-04-01T12:11:47Z","updated_at":"2015-04-12T22:26:34Z","closed_at":"2015-04-12T22:26:34Z","body":"Bug: confd doesn't try to reconnect to etcd cluster.\r\n\r\n\r\n```\r\n2015-04-01T12:07:46Z ip-10-105-20-115 /usr/local/bin/confd[29794]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-04-01T12:07:46Z ip-10-105-20-115 /usr/local/bin/confd[29794]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n```\r\n\r\nSteps to reproduce.\r\n\r\n1. Set up *broken* etcd cluster (in my example I had three etcd nodes that failed to connect to each other due to firewall rules)\r\n2. Start confd with watch=true, reading data from this broken etcd node.\r\n3. confd starts to report error 501 (as seen abobe)\r\n4. Fix etcd cluster (in my case: remove firewall rule that prevented etcd nodes from seeing each other)\r\n5. etcd is now healthy\r\n6. confd still reports error 501\r\n\r\nSeems like there is no retrying policy implemented in etcd connector (go-etcd?)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kelseyhightower/confd/issues/comments/92144764","html_url":"https://github.com/kelseyhightower/confd/issues/247#issuecomment-92144764","issue_url":"https://api.github.com/repos/kelseyhightower/confd/issues/247","id":92144764,"user":{"login":"zytek","id":3696278,"avatar_url":"https://avatars.githubusercontent.com/u/3696278?v=3","gravatar_id":"","url":"https://api.github.com/users/zytek","html_url":"https://github.com/zytek","followers_url":"https://api.github.com/users/zytek/followers","following_url":"https://api.github.com/users/zytek/following{/other_user}","gists_url":"https://api.github.com/users/zytek/gists{/gist_id}","starred_url":"https://api.github.com/users/zytek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zytek/subscriptions","organizations_url":"https://api.github.com/users/zytek/orgs","repos_url":"https://api.github.com/users/zytek/repos","events_url":"https://api.github.com/users/zytek/events{/privacy}","received_events_url":"https://api.github.com/users/zytek/received_events","type":"User","site_admin":false},"created_at":"2015-04-12T22:26:34Z","updated_at":"2015-04-12T22:26:34Z","body":"Well, while testing my patch I've found out that newest confd handles etcd failures properly, at least simple case of starting/stopping etcd localhost node (handling of \"connection refused\" errors) seems to work just fine. There is already a 2 second wait+retry code here:\r\n\r\nhttps://github.com/kelseyhightower/confd/blob/master/resource/template/processor.go#L94-L99\r\n\r\nI now think that my original test procedure was flawed and I failed to observe that. ;-) Closing this for now."}}