{"action":"created","issue":{"url":"https://api.github.com/repos/flapjack/flapjack/issues/915","labels_url":"https://api.github.com/repos/flapjack/flapjack/issues/915/labels{/name}","comments_url":"https://api.github.com/repos/flapjack/flapjack/issues/915/comments","events_url":"https://api.github.com/repos/flapjack/flapjack/issues/915/events","html_url":"https://github.com/flapjack/flapjack/issues/915","id":115182518,"number":915,"title":"Email pikelet redis connection wedges when redis goes down","user":{"login":"CVTJNII","id":13871930,"avatar_url":"https://avatars.githubusercontent.com/u/13871930?v=3","gravatar_id":"","url":"https://api.github.com/users/CVTJNII","html_url":"https://github.com/CVTJNII","followers_url":"https://api.github.com/users/CVTJNII/followers","following_url":"https://api.github.com/users/CVTJNII/following{/other_user}","gists_url":"https://api.github.com/users/CVTJNII/gists{/gist_id}","starred_url":"https://api.github.com/users/CVTJNII/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CVTJNII/subscriptions","organizations_url":"https://api.github.com/users/CVTJNII/orgs","repos_url":"https://api.github.com/users/CVTJNII/repos","events_url":"https://api.github.com/users/CVTJNII/events{/privacy}","received_events_url":"https://api.github.com/users/CVTJNII/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-05T00:33:02Z","updated_at":"2015-11-09T23:22:38Z","closed_at":null,"body":"Related to #886, I'm noticing that if the email pikelet fails to connect to Redis it will throw Redis::CannotConnectError indefinitely, even if Redis recovers.  The following stacktrace is logged:\r\n\r\n```\r\n2015-11-05T00:15:56.738044+00:00 [ERROR] :: flapjack-email :: Error generating or dispatching email message: Redis::CannotConnectError: Error connecting to Redis on tmhaproxy01.vagrant.tjnii.local:6379 (ECONNREFUSED)\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:290:in `rescue in establish_connection'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:285:in `establish_connection'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:79:in `block in connect'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:257:in `with_reconnect'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:78:in `connect'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:240:in `with_socket_timeout'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis/client.rb:178:in `call_with_timeout'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis.rb:1038:in `block in _bpop'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis.rb:37:in `block in synchronize'\r\n/usr/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis.rb:37:in `synchronize'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis.rb:1035:in `_bpop'\r\n/var/lib/gems/2.1.0/gems/redis-3.0.7/lib/redis.rb:1064:in `blpop'\r\n/var/lib/gems/2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/connection_pool.rb:81:in `block in method_missing'\r\n/var/lib/gems/2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/connection_pool.rb:25:in `execute'\r\n/var/lib/gems/2.1.0/gems/em-synchrony-1.0.4/lib/em-synchrony/connection_pool.rb:80:in `method_missing'\r\n/var/lib/gems/2.1.0/gems/flapjack-1.6.0/lib/flapjack/data/alert.rb:120:in `next'\r\n/var/lib/gems/2.1.0/gems/flapjack-1.6.0/lib/flapjack/gateways/email.rb:54:in `start'\r\n/var/lib/gems/2.1.0/gems/flapjack-1.6.0/lib/flapjack/pikelet.rb:130:in `block in start'\r\n```\r\n\r\nI've seen this exception repeating with Redis up.\r\n\r\nI have not found a reliable means to reproduce this yet, but it aligns with the behavior seen in #886 where Flapjack will emit this error until the logs fill the disk."},"comment":{"url":"https://api.github.com/repos/flapjack/flapjack/issues/comments/155230919","html_url":"https://github.com/flapjack/flapjack/issues/915#issuecomment-155230919","issue_url":"https://api.github.com/repos/flapjack/flapjack/issues/915","id":155230919,"user":{"login":"CVTJNII","id":13871930,"avatar_url":"https://avatars.githubusercontent.com/u/13871930?v=3","gravatar_id":"","url":"https://api.github.com/users/CVTJNII","html_url":"https://github.com/CVTJNII","followers_url":"https://api.github.com/users/CVTJNII/followers","following_url":"https://api.github.com/users/CVTJNII/following{/other_user}","gists_url":"https://api.github.com/users/CVTJNII/gists{/gist_id}","starred_url":"https://api.github.com/users/CVTJNII/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CVTJNII/subscriptions","organizations_url":"https://api.github.com/users/CVTJNII/orgs","repos_url":"https://api.github.com/users/CVTJNII/repos","events_url":"https://api.github.com/users/CVTJNII/events{/privacy}","received_events_url":"https://api.github.com/users/CVTJNII/received_events","type":"User","site_admin":false},"created_at":"2015-11-09T23:22:38Z","updated_at":"2015-11-09T23:22:38Z","body":"This is larger than the email pikelet, though that's the noisiest in the logs.  I just had this happen due to a transient network issue causing Sentinel to promote a Redis slave to master, causing the connection to be interrupted as haproxy transitioned.  At the same time I began seeing these logs and the API went unavailable with flapjack-diner throwing '/usr/lib/ruby/2.1.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)'.\r\n\r\nIn our environment this bug is getting nasty, I'm going to have to try to root cause and squish it.  Any advice would be appreciated as I've I haven't done any deep-dives into the Flapjack code yet."}}