{"action":"opened","issue":{"url":"https://api.github.com/repos/BlueDragonX/sentinel/issues/7","labels_url":"https://api.github.com/repos/BlueDragonX/sentinel/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/BlueDragonX/sentinel/issues/7/comments","events_url":"https://api.github.com/repos/BlueDragonX/sentinel/issues/7/events","html_url":"https://github.com/BlueDragonX/sentinel/issues/7","id":54325384,"number":7,"title":"Sentinel executes templates even when it can't reach etcd","user":{"login":"anoldguy","id":1304498,"avatar_url":"https://avatars.githubusercontent.com/u/1304498?v=3","gravatar_id":"","url":"https://api.github.com/users/anoldguy","html_url":"https://github.com/anoldguy","followers_url":"https://api.github.com/users/anoldguy/followers","following_url":"https://api.github.com/users/anoldguy/following{/other_user}","gists_url":"https://api.github.com/users/anoldguy/gists{/gist_id}","starred_url":"https://api.github.com/users/anoldguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anoldguy/subscriptions","organizations_url":"https://api.github.com/users/anoldguy/orgs","repos_url":"https://api.github.com/users/anoldguy/repos","events_url":"https://api.github.com/users/anoldguy/events{/privacy}","received_events_url":"https://api.github.com/users/anoldguy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-14T14:05:49Z","updated_at":"2015-01-14T14:05:49Z","closed_at":null,"body":"Hi Ryan!\r\n\r\nWe've noticed our config files getting overwritten when sentinel can't contact the etcd servers.  Specifically, this is biting us at boot time.  Here's what we're seeing.  Our files are being modified right when we reboot our box, and get overwritten with the empty template.\r\n\r\n```\r\nroot@host:/path/to/app/config# ll\r\ndrwxr-xr-x  2 user group 4096 Jan 13 23:30 ./\r\ndrwxr-xr-x 13 user group 4096 Nov 28 21:38 ../\r\n-rw-------  1 user group   69 Jan 13 23:30 database_config.yml\r\n-rw-------  1 user group   69 Jan 13 23:30 memcached_config.yml\r\nroot@host:/path/to/app/config# cat memcached_config.yml\r\n# This is a sentinel-generated config file!\r\n# Do not edit by hand!\r\n\r\nroot@host:/path/to/app/config#\r\n```\r\nThe last modified time corresponds to the sentinel start times:\r\n\r\n```\r\nroot@host:/path/to/app/config# ps axuwwf | grep sentinel\r\nuser        651  0.0  0.0 399556  4792 ?        Ssl  23:30   0:00 /usr/local/bin/sentinel -config=/etc/sentinel.d/app4_staging.yml\r\nuser        656  0.0  0.0 399556  4672 ?        Ssl  23:30   0:00 /usr/local/bin/sentinel -config=/etc/sentinel.d/app3_staging.yml\r\nuser        657  0.0  0.0 399684  4668 ?        Ssl  23:30   0:00 /usr/local/bin/sentinel -config=/etc/sentinel.d/app2_staging.yml\r\nuser        658  0.0  0.0 399684  4760 ?        Ssl  23:30   0:00 /usr/local/bin/sentinel -config=/etc/sentinel.d/app1_staging.yml\r\n```\r\n\r\nWe've got our sentinels configured via upstart, and I noticed it is starting sentinel before the network is available.  Fair enough. So we modify our upstart config to wait until the specific interface we need is up and ready.  No dice. It turns out that udev tells upstart the interface is up about 10s before the bond is able to send any traffic.  So we go lo-fi and put a sleep in place. Yuk.  However, it works, and sentinel writes correct configs.\r\n\r\nHowever, I'm proposing that it is incorrect behavior to execute the templates for the watchers if you can't talk to etcd.\r\n\r\nIn the sentinel logs (during another reboot test), we see:\r\n```\r\nsentinel 2015/01/14 04:40:54.605180 [DEBUG]  get key '/shared/app1/staging/memcached': 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]\r\nsentinel 2015/01/14 04:40:54.605192 [DEBUG]  context: map[]\r\n\r\nsentinel 2015/01/14 04:40:54.609909 [DEBUG]  template '/etc/sentinel.d/app1_staging/generic.tpl' rendered to '/path/to/app/config/memcached.yml'\r\nsentinel 2015/01/14 04:40:54.609926 [DEBUG]  app1_memcached has no command, skipping\r\nsentinel 2015/01/14 04:40:54.609932 [INFO]   app1_memcached executed\r\n```\r\nThis implied we had an issue with our etcd setup, so we started investigating that, but it turns out tracking down this error message leads me to the [go-etcd package](https://github.com/coreos/go-etcd/blob/master/etcd/error.go#L8-10). So from sentinel's standpoint, it got a 501 from etcd, but still executed the template with an empty context, which, of course, was written to disk, as the result was different.\r\n\r\nI do not know golang, but I think if there were a check in the [watcher#Run function](https://github.com/BlueDragonX/sentinel/blob/master/watcher.go#L106-115), that would inspect the event and see if it were empty, or maybe in [client#GetMaps function](https://github.com/BlueDragonX/sentinel/blob/master/client.go#L110-121), detect if the map is empty and return an error. I don't know.\r\n\r\nHowever, I do think it's a bug that sentinel re-renders the templates when it can't contact etcd."}}