{"action":"opened","issue":{"url":"https://api.github.com/repos/grafana/grafana/issues/2989","labels_url":"https://api.github.com/repos/grafana/grafana/issues/2989/labels{/name}","comments_url":"https://api.github.com/repos/grafana/grafana/issues/2989/comments","events_url":"https://api.github.com/repos/grafana/grafana/issues/2989/events","html_url":"https://github.com/grafana/grafana/issues/2989","id":112474381,"number":2989,"title":"email notifications hanging when connection times out","user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-20T22:36:34Z","updated_at":"2015-10-20T22:36:34Z","closed_at":null,"body":"In the raintank/grafana server i was trying to debug why the alerting executors seemed to hang occasionally and not process new jobs.\r\nI think the problem is also in grafana/grafana.\r\nI saw this message in logs\r\n```\r\n2015/10/20 20:46:41 ESC[1;31m[asm_amd64.s:2232 goexit()] [E] Async sent email 0 succeed, not send emails:  err: dial tcp 104.130.177.23:25: connection timed outESC[0m\r\n2015/10/20 20:46:41 ESC[1;36m[T] Sending mails to: ESC[0m\r\n\r\n```\r\ni got suspicious that the timeout might be way too long, so i did a full goroutine dump at https://gist.github.com/Dieterbe/77a5b82aa97ea0ce2cc5\r\n\r\nI found 10 cases of notifications.go sendEmailCommandHandler's addToMailQueue not able to send to mailQueue because it's blocked from taking in new entries\r\n\r\n```\r\ndieter@dieter-m6800 ~ grep -B 2 'notifications/mailer.go:47' dump.txt \r\ngoroutine 68 [chan send, 10 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d20f260)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 69 [chan send, 6 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d20e9c0)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 70 [chan send, 10 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d0a9620)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 71 [chan send, 2 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc208165c80)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 72 [chan send, 10 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d0a58c0)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 73 [chan send, 8 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d0a4000)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 74 [chan send, 4 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc208b5c060)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 75 [chan send, 10 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d18b9e0)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 76 [chan send]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d0a5c20)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n--\r\ngoroutine 77 [chan send, 10 minutes]:\r\ngithub.com/grafana/grafana/pkg/services/notifications.func·001(0xc20d047c20)\r\n\t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:47 +0x4a\r\n```\r\n\r\ngoroutine 78 is blocked on iowait for `conn, err := net.Dial(\"tcp\", net.JoinHostPort(host, port))`\r\nthough it's not clear for how long, and i also don't understand yet why this code path only shows up once instead of 10 times.\r\nwill do some more research"}}