{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/2895","labels_url":"https://api.github.com/repos/deis/deis/issues/2895/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/2895/comments","events_url":"https://api.github.com/repos/deis/deis/issues/2895/events","html_url":"https://github.com/deis/deis/issues/2895","id":54310681,"number":2895,"title":"cluster is working, but filling logs with confd errors","user":{"login":"jannispl","id":838818,"avatar_url":"https://avatars.githubusercontent.com/u/838818?v=3","gravatar_id":"","url":"https://api.github.com/users/jannispl","html_url":"https://github.com/jannispl","followers_url":"https://api.github.com/users/jannispl/followers","following_url":"https://api.github.com/users/jannispl/following{/other_user}","gists_url":"https://api.github.com/users/jannispl/gists{/gist_id}","starred_url":"https://api.github.com/users/jannispl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jannispl/subscriptions","organizations_url":"https://api.github.com/users/jannispl/orgs","repos_url":"https://api.github.com/users/jannispl/repos","events_url":"https://api.github.com/users/jannispl/events{/privacy}","received_events_url":"https://api.github.com/users/jannispl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-01-14T11:16:00Z","updated_at":"2015-01-14T16:37:51Z","closed_at":null,"body":"After upgrading my 1.1.1 staging cluster to 1.2.0, my cluster seems to be in a working state with all units on `active/running`. However, I noticed that some units are filling my logs with confd errors.\r\n\r\ndeis-router:\r\n```\r\n[INFO] - 2015/01/14 11:02:46 [notice] 154#0: signal process started\r\n[INFO] - 2015/01/14 11:02:51 [notice] 164#0: signal process started\r\n[WARNING] - 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-01-14T11:03:20Z 01e304387886 confd[44]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-01-14T11:03:21Z 01e304387886 confd[44]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n[INFO] - 2015/01/14 11:03:31 [notice] 174#0: signal process started\r\n[INFO] - 2015/01/14 11:03:37 [notice] 184#0: signal process started\r\n2015-01-14T11:03:45Z 01e304387886 confd[44]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-01-14T11:03:45Z 01e304387886 confd[44]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-01-14T11:03:45Z 01e304387886 confd[44]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n```\r\n\r\ndeis-builder:\r\n```\r\n2015-01-14T11:05:28Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:05:28Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:05:28Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n[32094c8a] -job pull(qnsolutions/appspark-web-base, v22) = OK (0)\r\n2015-01-14T11:05:44Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:05:44Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:05:44Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n[32094c8a] +job allocate_interface(b7b25be9d28996a2b29d00845ebff0c9011520bf4e9fd8c867c7ec6b94e577d5)\r\n[32094c8a] -job allocate_interface(b7b25be9d28996a2b29d00845ebff0c9011520bf4e9fd8c867c7ec6b94e577d5) = OK (0)\r\n[32094c8a] +job log(start, b7b25be9d28996a2b29d00845ebff0c9011520bf4e9fd8c867c7ec6b94e577d5, 30421519a995)\r\n[32094c8a] -job log(start, b7b25be9d28996a2b29d00845ebff0c9011520bf4e9fd8c867c7ec6b94e577d5, 30421519a995) = OK (0)\r\n2015-01-14T11:06:09Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:06:09Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:06:09Z a4d63487fc6c confd[146]: ERROR Config check failed: exit status 1\r\n```\r\n\r\ndeis-controller:\r\n```\r\n2015-01-14T11:07:51Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:08:07Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:08:14Z fd5ae229b1e9 confd[93]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\nError:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\nError:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\n2015-01-14T11:08:19Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:08:24Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:08:39Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n2015-01-14T11:08:59Z fd5ae229b1e9 confd[93]: ERROR Config check failed: exit status 1\r\n```\r\n\r\n`/opt/bin/deis-debug-logs`:\r\n```\r\n$ /opt/bin/deis-debug-logs \r\n--- VERSIONS ---\r\nCoreOS 509.1.0\r\nDeis v1.0.2\r\netcd version 0.4.6\r\nfleet version 0.8.3\r\n\r\n--- SYSTEM STATUS ---\r\n\r\n(freezes here)\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/69945753","html_url":"https://github.com/deis/deis/issues/2895#issuecomment-69945753","issue_url":"https://api.github.com/repos/deis/deis/issues/2895","id":69945753,"user":{"login":"bacongobbler","id":1360539,"avatar_url":"https://avatars.githubusercontent.com/u/1360539?v=3","gravatar_id":"","url":"https://api.github.com/users/bacongobbler","html_url":"https://github.com/bacongobbler","followers_url":"https://api.github.com/users/bacongobbler/followers","following_url":"https://api.github.com/users/bacongobbler/following{/other_user}","gists_url":"https://api.github.com/users/bacongobbler/gists{/gist_id}","starred_url":"https://api.github.com/users/bacongobbler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bacongobbler/subscriptions","organizations_url":"https://api.github.com/users/bacongobbler/orgs","repos_url":"https://api.github.com/users/bacongobbler/repos","events_url":"https://api.github.com/users/bacongobbler/events{/privacy}","received_events_url":"https://api.github.com/users/bacongobbler/received_events","type":"User","site_admin":false},"created_at":"2015-01-14T16:37:51Z","updated_at":"2015-01-14T16:37:51Z","body":"According to https://github.com/coreos/etcd/issues/863#issuecomment-60523183 you may be in an unrecoverable state if you brought a majority of the machines down. I'd take a look at that issue and see if there's something that may be useful."}}