{"action":"created","comment":{"url":"https://api.github.com/repos/deis/deis/pulls/comments/41544105","id":41544105,"diff_hunk":"@@ -0,0 +1,117 @@\n+package simple\n+\n+import (\n+\t\"fmt\"\n+\t\"log\"\n+\t\"net\"\n+\t\"net/url\"\n+\t\"sync\"\n+\t\"time\"\n+)\n+\n+// This determines how often we'll replace an existing connection with a new one.  This ensures we\n+// redial, which in turn will ensure any DNS changes affecting where logs are drained to don't\n+// go unnoticed for too long.\n+const connRefreshInterval = 5 * time.Minute\n+\n+// This determines how many failed dial attempts are required before the drain is muted.\n+const maxFailedConns = 5\n+\n+// This determines how much time we're willing to spend dialing.\n+const dialTimeout = 10 * time.Second\n+\n+// This is how long the drain is muted for after repeated connection failures.\n+const mutePeriod = 5 * time.Minute\n+\n+type logDrain struct {\n+\tproto    string\n+\turi      string\n+\tconn     net.Conn\n+\tconnTime time.Time\n+\tmuted    bool\n+\tmutex    sync.Mutex\n+}\n+\n+// NewDrain returns a pointer to a new instance of a drain.LogDrain\n+func NewDrain(drainURL string) (*logDrain, error) {\n+\tu, err := url.Parse(drainURL)\n+\tif err != nil {\n+\t\treturn nil, err\n+\t}\n+\tvar proto string\n+\tif u.Scheme == \"udp\" || u.Scheme == \"syslog\" {\n+\t\tproto = \"udp\"\n+\t} else if u.Scheme == \"tcp\" {\n+\t\tproto = \"tcp\"\n+\t} else {\n+\t\treturn nil, fmt.Errorf(\"Invalid drain url scheme: %s\", u.Scheme)\n+\t}\n+\treturn &logDrain{proto: proto, uri: u.Host + u.Path}, nil\n+}\n+\n+// Send forwards the provided log message to an external destination\n+func (d *logDrain) Send(message string) error {\n+\tif d.muted {\n+\t\treturn nil\n+\t}\n+\td.mutex.Lock()\n+\tdefer d.mutex.Unlock()\n+\tconn, err := d.getConnection(false)\n+\tif err != nil {\n+\t\treturn err\n+\t}\n+\t_, err = fmt.Fprintln(conn, message)\n+\tif err != nil {","path":"logger/drain/simple/drain.go","position":64,"original_position":64,"commit_id":"d28e477b7ab82616244e76aad74f1b5eec22a292","original_commit_id":"d28e477b7ab82616244e76aad74f1b5eec22a292","user":{"login":"krancour","id":1821014,"avatar_url":"https://avatars.githubusercontent.com/u/1821014?v=3","gravatar_id":"","url":"https://api.github.com/users/krancour","html_url":"https://github.com/krancour","followers_url":"https://api.github.com/users/krancour/followers","following_url":"https://api.github.com/users/krancour/following{/other_user}","gists_url":"https://api.github.com/users/krancour/gists{/gist_id}","starred_url":"https://api.github.com/users/krancour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krancour/subscriptions","organizations_url":"https://api.github.com/users/krancour/orgs","repos_url":"https://api.github.com/users/krancour/repos","events_url":"https://api.github.com/users/krancour/events{/privacy}","received_events_url":"https://api.github.com/users/krancour/received_events","type":"User","site_admin":false},"body":"I think this is another thing to come back to if there's a demonstrated need for it.\r\n\r\nCurrently, there are two things that might fail and thus be retried.  A failed message write is probably on account of a broken pipe.  It's retried with a new connection.  Assuming the new connection was successfully obtained, there's no reason the second attempt shouldn't succeed.  So on that front, I'm not sure if we need to get any fancier... at least for now.\r\n\r\nThe other thing that gets retried are failed dialing attempts.  That's currently hardcoded through a constant to be retried five times before giving up, at which point there isn't a prayer of firing off a log message to the external service, so the drain is temporarily muted.\r\n\r\nMuting a log drain has the benefit of (forgive the overloaded term here) _draining_ the internal message queue very quickly so that when things recover, we're draining recent log messages out to the external log archive instead of working through a backlog of older log messages while newer log messages get dropped because of the backed up queue.\r\n\r\nI _think_ this all sounds pretty reasonable, but I am more than willing to hear further challenges to that.","created_at":"2015-10-08T17:42:04Z","updated_at":"2015-10-08T17:42:04Z","html_url":"https://github.com/deis/deis/pull/4578#discussion_r41544105","pull_request_url":"https://api.github.com/repos/deis/deis/pulls/4578","_links":{"self":{"href":"https://api.github.com/repos/deis/deis/pulls/comments/41544105"},"html":{"href":"https://github.com/deis/deis/pull/4578#discussion_r41544105"},"pull_request":{"href":"https://api.github.com/repos/deis/deis/pulls/4578"}}},"pull_request":{"url":"https://api.github.com/repos/deis/deis/pulls/4578","id":47111695,"html_url":"https://github.com/deis/deis/pull/4578","diff_url":"https://github.com/deis/deis/pull/4578.diff","patch_url":"https://github.com/deis/deis/pull/4578.patch","issue_url":"https://api.github.com/repos/deis/deis/issues/4578","number":4578,"state":"open","locked":false,"title":"chore(logger): improve drains","user":{"login":"krancour","id":1821014,"avatar_url":"https://avatars.githubusercontent.com/u/1821014?v=3","gravatar_id":"","url":"https://api.github.com/users/krancour","html_url":"https://github.com/krancour","followers_url":"https://api.github.com/users/krancour/followers","following_url":"https://api.github.com/users/krancour/following{/other_user}","gists_url":"https://api.github.com/users/krancour/gists{/gist_id}","starred_url":"https://api.github.com/users/krancour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krancour/subscriptions","organizations_url":"https://api.github.com/users/krancour/orgs","repos_url":"https://api.github.com/users/krancour/repos","events_url":"https://api.github.com/users/krancour/events{/privacy}","received_events_url":"https://api.github.com/users/krancour/received_events","type":"User","site_admin":false},"body":"This replaces the existing \"good enough\" UDP-based and TCP-based log drains with one drain that can accommodate both and, more importantly, is more performant and resilient.\r\n\r\nCliff notes:\r\n\r\n* It reuses connections for up to 5 minutes, but not more than that.  This is to increase efficiency by dialing less, while still dialing often enough that any DNS changes that affect delivery of messages to an external log archive won't go unnoticed for very long.\r\n* It can detect a broken pipe and try resending a message after establishing a new connection.\r\n* Repeated failed dialing attempts will \"mute\" the drain for five minutes.  This gives the network a break (necessary, especially if the failures may be due to any sort of external DNS or external log service rate limiting) and allows the backed up message queue to be cleared more quickly.  If a drain is muted for any period of time, it is noted in the logger's own journal logs.","created_at":"2015-10-08T04:54:49Z","updated_at":"2015-10-08T17:42:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"64bfa178f1087ea949980b1453dbdf78c9220ffa","assignee":{"login":"krancour","id":1821014,"avatar_url":"https://avatars.githubusercontent.com/u/1821014?v=3","gravatar_id":"","url":"https://api.github.com/users/krancour","html_url":"https://github.com/krancour","followers_url":"https://api.github.com/users/krancour/followers","following_url":"https://api.github.com/users/krancour/following{/other_user}","gists_url":"https://api.github.com/users/krancour/gists{/gist_id}","starred_url":"https://api.github.com/users/krancour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krancour/subscriptions","organizations_url":"https://api.github.com/users/krancour/orgs","repos_url":"https://api.github.com/users/krancour/repos","events_url":"https://api.github.com/users/krancour/events{/privacy}","received_events_url":"https://api.github.com/users/krancour/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/deis/deis/milestones/46","html_url":"https://github.com/deis/deis/milestones/v1.12","labels_url":"https://api.github.com/repos/deis/deis/milestones/46/labels","id":1338456,"number":46,"title":"v1.12","description":"","creator":{"login":"mboersma","id":73019,"avatar_url":"https://avatars.githubusercontent.com/u/73019?v=3","gravatar_id":"","url":"https://api.github.com/users/mboersma","html_url":"https://github.com/mboersma","followers_url":"https://api.github.com/users/mboersma/followers","following_url":"https://api.github.com/users/mboersma/following{/other_user}","gists_url":"https://api.github.com/users/mboersma/gists{/gist_id}","starred_url":"https://api.github.com/users/mboersma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mboersma/subscriptions","organizations_url":"https://api.github.com/users/mboersma/orgs","repos_url":"https://api.github.com/users/mboersma/repos","events_url":"https://api.github.com/users/mboersma/events{/privacy}","received_events_url":"https://api.github.com/users/mboersma/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":3,"state":"open","created_at":"2015-10-05T15:12:42Z","updated_at":"2015-10-08T16:28:54Z","due_on":"2015-11-03T07:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/deis/deis/pulls/4578/commits","review_comments_url":"https://api.github.com/repos/deis/deis/pulls/4578/comments","review_comment_url":"https://api.github.com/repos/deis/deis/pulls/comments{/number}","comments_url":"https://api.github.com/repos/deis/deis/issues/4578/comments","statuses_url":"https://api.github.com/repos/deis/deis/statuses/d28e477b7ab82616244e76aad74f1b5eec22a292","head":{"label":"krancour:improve-drains","ref":"improve-drains","sha":"d28e477b7ab82616244e76aad74f1b5eec22a292","user":{"login":"krancour","id":1821014,"avatar_url":"https://avatars.githubusercontent.com/u/1821014?v=3","gravatar_id":"","url":"https://api.github.com/users/krancour","html_url":"https://github.com/krancour","followers_url":"https://api.github.com/users/krancour/followers","following_url":"https://api.github.com/users/krancour/following{/other_user}","gists_url":"https://api.github.com/users/krancour/gists{/gist_id}","starred_url":"https://api.github.com/users/krancour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krancour/subscriptions","organizations_url":"https://api.github.com/users/krancour/orgs","repos_url":"https://api.github.com/users/krancour/repos","events_url":"https://api.github.com/users/krancour/events{/privacy}","received_events_url":"https://api.github.com/users/krancour/received_events","type":"User","site_admin":false},"repo":{"id":33512149,"name":"deis","full_name":"krancour/deis","owner":{"login":"krancour","id":1821014,"avatar_url":"https://avatars.githubusercontent.com/u/1821014?v=3","gravatar_id":"","url":"https://api.github.com/users/krancour","html_url":"https://github.com/krancour","followers_url":"https://api.github.com/users/krancour/followers","following_url":"https://api.github.com/users/krancour/following{/other_user}","gists_url":"https://api.github.com/users/krancour/gists{/gist_id}","starred_url":"https://api.github.com/users/krancour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krancour/subscriptions","organizations_url":"https://api.github.com/users/krancour/orgs","repos_url":"https://api.github.com/users/krancour/repos","events_url":"https://api.github.com/users/krancour/events{/privacy}","received_events_url":"https://api.github.com/users/krancour/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/krancour/deis","description":"Your PaaS. Your Rules. ","fork":true,"url":"https://api.github.com/repos/krancour/deis","forks_url":"https://api.github.com/repos/krancour/deis/forks","keys_url":"https://api.github.com/repos/krancour/deis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/krancour/deis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/krancour/deis/teams","hooks_url":"https://api.github.com/repos/krancour/deis/hooks","issue_events_url":"https://api.github.com/repos/krancour/deis/issues/events{/number}","events_url":"https://api.github.com/repos/krancour/deis/events","assignees_url":"https://api.github.com/repos/krancour/deis/assignees{/user}","branches_url":"https://api.github.com/repos/krancour/deis/branches{/branch}","tags_url":"https://api.github.com/repos/krancour/deis/tags","blobs_url":"https://api.github.com/repos/krancour/deis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/krancour/deis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/krancour/deis/git/refs{/sha}","trees_url":"https://api.github.com/repos/krancour/deis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/krancour/deis/statuses/{sha}","languages_url":"https://api.github.com/repos/krancour/deis/languages","stargazers_url":"https://api.github.com/repos/krancour/deis/stargazers","contributors_url":"https://api.github.com/repos/krancour/deis/contributors","subscribers_url":"https://api.github.com/repos/krancour/deis/subscribers","subscription_url":"https://api.github.com/repos/krancour/deis/subscription","commits_url":"https://api.github.com/repos/krancour/deis/commits{/sha}","git_commits_url":"https://api.github.com/repos/krancour/deis/git/commits{/sha}","comments_url":"https://api.github.com/repos/krancour/deis/comments{/number}","issue_comment_url":"https://api.github.com/repos/krancour/deis/issues/comments{/number}","contents_url":"https://api.github.com/repos/krancour/deis/contents/{+path}","compare_url":"https://api.github.com/repos/krancour/deis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/krancour/deis/merges","archive_url":"https://api.github.com/repos/krancour/deis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/krancour/deis/downloads","issues_url":"https://api.github.com/repos/krancour/deis/issues{/number}","pulls_url":"https://api.github.com/repos/krancour/deis/pulls{/number}","milestones_url":"https://api.github.com/repos/krancour/deis/milestones{/number}","notifications_url":"https://api.github.com/repos/krancour/deis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/krancour/deis/labels{/name}","releases_url":"https://api.github.com/repos/krancour/deis/releases{/id}","created_at":"2015-04-06T23:58:08Z","updated_at":"2015-05-05T14:10:36Z","pushed_at":"2015-10-08T04:45:08Z","git_url":"git://github.com/krancour/deis.git","ssh_url":"git@github.com:krancour/deis.git","clone_url":"https://github.com/krancour/deis.git","svn_url":"https://github.com/krancour/deis","homepage":"http://deis.io","size":20010,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"deis:master","ref":"master","sha":"1149eb5c043e29e20b798672b262398c4332c1e4","user":{"login":"deis","id":6117407,"avatar_url":"https://avatars.githubusercontent.com/u/6117407?v=3","gravatar_id":"","url":"https://api.github.com/users/deis","html_url":"https://github.com/deis","followers_url":"https://api.github.com/users/deis/followers","following_url":"https://api.github.com/users/deis/following{/other_user}","gists_url":"https://api.github.com/users/deis/gists{/gist_id}","starred_url":"https://api.github.com/users/deis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deis/subscriptions","organizations_url":"https://api.github.com/users/deis/orgs","repos_url":"https://api.github.com/users/deis/repos","events_url":"https://api.github.com/users/deis/events{/privacy}","received_events_url":"https://api.github.com/users/deis/received_events","type":"Organization","site_admin":false},"repo":{"id":11589713,"name":"deis","full_name":"deis/deis","owner":{"login":"deis","id":6117407,"avatar_url":"https://avatars.githubusercontent.com/u/6117407?v=3","gravatar_id":"","url":"https://api.github.com/users/deis","html_url":"https://github.com/deis","followers_url":"https://api.github.com/users/deis/followers","following_url":"https://api.github.com/users/deis/following{/other_user}","gists_url":"https://api.github.com/users/deis/gists{/gist_id}","starred_url":"https://api.github.com/users/deis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deis/subscriptions","organizations_url":"https://api.github.com/users/deis/orgs","repos_url":"https://api.github.com/users/deis/repos","events_url":"https://api.github.com/users/deis/events{/privacy}","received_events_url":"https://api.github.com/users/deis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/deis/deis","description":"Your PaaS. Your Rules. ","fork":false,"url":"https://api.github.com/repos/deis/deis","forks_url":"https://api.github.com/repos/deis/deis/forks","keys_url":"https://api.github.com/repos/deis/deis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deis/deis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deis/deis/teams","hooks_url":"https://api.github.com/repos/deis/deis/hooks","issue_events_url":"https://api.github.com/repos/deis/deis/issues/events{/number}","events_url":"https://api.github.com/repos/deis/deis/events","assignees_url":"https://api.github.com/repos/deis/deis/assignees{/user}","branches_url":"https://api.github.com/repos/deis/deis/branches{/branch}","tags_url":"https://api.github.com/repos/deis/deis/tags","blobs_url":"https://api.github.com/repos/deis/deis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deis/deis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deis/deis/git/refs{/sha}","trees_url":"https://api.github.com/repos/deis/deis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deis/deis/statuses/{sha}","languages_url":"https://api.github.com/repos/deis/deis/languages","stargazers_url":"https://api.github.com/repos/deis/deis/stargazers","contributors_url":"https://api.github.com/repos/deis/deis/contributors","subscribers_url":"https://api.github.com/repos/deis/deis/subscribers","subscription_url":"https://api.github.com/repos/deis/deis/subscription","commits_url":"https://api.github.com/repos/deis/deis/commits{/sha}","git_commits_url":"https://api.github.com/repos/deis/deis/git/commits{/sha}","comments_url":"https://api.github.com/repos/deis/deis/comments{/number}","issue_comment_url":"https://api.github.com/repos/deis/deis/issues/comments{/number}","contents_url":"https://api.github.com/repos/deis/deis/contents/{+path}","compare_url":"https://api.github.com/repos/deis/deis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deis/deis/merges","archive_url":"https://api.github.com/repos/deis/deis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deis/deis/downloads","issues_url":"https://api.github.com/repos/deis/deis/issues{/number}","pulls_url":"https://api.github.com/repos/deis/deis/pulls{/number}","milestones_url":"https://api.github.com/repos/deis/deis/milestones{/number}","notifications_url":"https://api.github.com/repos/deis/deis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deis/deis/labels{/name}","releases_url":"https://api.github.com/repos/deis/deis/releases{/id}","created_at":"2013-07-22T19:02:40Z","updated_at":"2015-10-08T15:36:03Z","pushed_at":"2015-10-08T17:32:07Z","git_url":"git://github.com/deis/deis.git","ssh_url":"git@github.com:deis/deis.git","clone_url":"https://github.com/deis/deis.git","svn_url":"https://github.com/deis/deis","homepage":"http://deis.io","size":43209,"stargazers_count":4819,"watchers_count":4819,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":762,"mirror_url":null,"open_issues_count":304,"forks":762,"open_issues":304,"watchers":4819,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/deis/deis/pulls/4578"},"html":{"href":"https://github.com/deis/deis/pull/4578"},"issue":{"href":"https://api.github.com/repos/deis/deis/issues/4578"},"comments":{"href":"https://api.github.com/repos/deis/deis/issues/4578/comments"},"review_comments":{"href":"https://api.github.com/repos/deis/deis/pulls/4578/comments"},"review_comment":{"href":"https://api.github.com/repos/deis/deis/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/deis/deis/pulls/4578/commits"},"statuses":{"href":"https://api.github.com/repos/deis/deis/statuses/d28e477b7ab82616244e76aad74f1b5eec22a292"}}}}