{"action":"opened","number":601,"pull_request":{"url":"https://api.github.com/repos/bitly/nsq/pulls/601","id":39872040,"html_url":"https://github.com/bitly/nsq/pull/601","diff_url":"https://github.com/bitly/nsq/pull/601.diff","patch_url":"https://github.com/bitly/nsq/pull/601.patch","issue_url":"https://api.github.com/repos/bitly/nsq/issues/601","number":601,"state":"open","locked":false,"title":"nsqd: ability to dynamically reconfigure nsqlookupd addresses","user":{"login":"mreiferson","id":187441,"avatar_url":"https://avatars.githubusercontent.com/u/187441?v=3","gravatar_id":"","url":"https://api.github.com/users/mreiferson","html_url":"https://github.com/mreiferson","followers_url":"https://api.github.com/users/mreiferson/followers","following_url":"https://api.github.com/users/mreiferson/following{/other_user}","gists_url":"https://api.github.com/users/mreiferson/gists{/gist_id}","starred_url":"https://api.github.com/users/mreiferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreiferson/subscriptions","organizations_url":"https://api.github.com/users/mreiferson/orgs","repos_url":"https://api.github.com/users/mreiferson/repos","events_url":"https://api.github.com/users/mreiferson/events{/privacy}","received_events_url":"https://api.github.com/users/mreiferson/received_events","type":"User","site_admin":false},"body":"I'd love it if NSQ could support DNS-based discovery for nsqlookupd instances. This would be a light layer on top of the already-proven architecture that would allow ops people to abstract away some of the configuration required to get NSQ working.\r\n\r\nBasically, rather than hardcoding nsqlookupd addresses, I'd much rather just say \"use all the A records returned by this DNS name\". There's also an [RFC](http://www.ietf.org/rfc/rfc6763.txt) for using DNS for service discovery. This supports things like [Prometheus](http://prometheus.io) and makes deployment a snap on [Kubernetes](http://kubernetes.io). It basically makes it dead simple to use NSQ in a cluster environment, where you don't _know_ what IP address or machine any of your services will be available at, but know you can reach them through a DNS lookup.\r\n\r\n[More docs](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/services.md#headless-services) on the relevant portion of Kubernetes.\r\n\r\nHere's the desired user experience:\r\n\r\n1. I set up a service (logical grouping) of pods (group of containers, think \"application cookie cutter\") and say I want 4 of them. Kubernetes automatically _does the right thing_ for me, setting them up, tearing them down, restarting them, moving them, whatever.\r\n2. I configure nsqd to say \"do a DNS lookup on nsqlookupd.cluster, and connect to every IP address returned as an A record. Also, every minute/five minutes/ten minutes, check again, and update your connections\"\r\n3. I configure my consumers to say \"do a DNS lookup on nsqlookupd.cluster, and connect to every IP address returned as an A record. Also, every minute/five minutes/ten minutes, check again and update your connections\"\r\n4. Everything just works. Containers get restarted, machines get removed, machines get added, containers get moved, and everything just points to the right thing.\r\n\r\nMost of this (the DNS querying part) can be achieved simply with wrappers/helpers. The big missing piece is some sort of runtime live-reloading/editing of configuration. Any of the following would work beautifully:\r\n\r\n* A way to easily wrap nsqd inside another Go binary, and use a channel/function/whatever to say \"reload configuration/here are new configuration values\"\r\n* A way to say \"Here's a configuration file, read the nsqlookupd addresses out of it, and re-check whenever I HUP you\" (I'm unsure whether one container can HUP a process in another. If not, this is less ideal, but still probably workable. It'll just take some noodling on how to make it happen. The unix socket approach would be more desirable, honestly)\r\n* A TCP/unix socket/http/whatever interface that we can use to pass new configuration values into.\r\n\r\nOr I'm sure there are myriad better ways to accomplish this. Basically, the important bit is \"here are the new nsqlookupd addresses, use them instead, but don't stop running/serving requests, and please don't catch on fire\".\r\n\r\nI'm making the following assumptions when imagining how this would be run:\r\n\r\n* A fresh nsqlookupd will be able to gather shared state from the registering consumers/nsqd instances quickly-ish (maybe a few seconds, at most?)\r\n* Assuming each nsqd/consumer connects to at least two nsqlookupd instances, one going away doesn't actually affect anything except your redundancy level.","created_at":"2015-07-14T00:46:18Z","updated_at":"2015-07-14T00:46:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/bitly/nsq/pulls/601/commits","review_comments_url":"https://api.github.com/repos/bitly/nsq/pulls/601/comments","review_comment_url":"https://api.github.com/repos/bitly/nsq/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bitly/nsq/issues/601/comments","statuses_url":"https://api.github.com/repos/bitly/nsq/statuses/21ef3fe7ed46317ef266d7e0c95b28eae498cd53","head":{"label":"mreiferson:nsqd_config_endpoint_601","ref":"nsqd_config_endpoint_601","sha":"21ef3fe7ed46317ef266d7e0c95b28eae498cd53","user":{"login":"mreiferson","id":187441,"avatar_url":"https://avatars.githubusercontent.com/u/187441?v=3","gravatar_id":"","url":"https://api.github.com/users/mreiferson","html_url":"https://github.com/mreiferson","followers_url":"https://api.github.com/users/mreiferson/followers","following_url":"https://api.github.com/users/mreiferson/following{/other_user}","gists_url":"https://api.github.com/users/mreiferson/gists{/gist_id}","starred_url":"https://api.github.com/users/mreiferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreiferson/subscriptions","organizations_url":"https://api.github.com/users/mreiferson/orgs","repos_url":"https://api.github.com/users/mreiferson/repos","events_url":"https://api.github.com/users/mreiferson/events{/privacy}","received_events_url":"https://api.github.com/users/mreiferson/received_events","type":"User","site_admin":false},"repo":{"id":4343373,"name":"nsq","full_name":"mreiferson/nsq","owner":{"login":"mreiferson","id":187441,"avatar_url":"https://avatars.githubusercontent.com/u/187441?v=3","gravatar_id":"","url":"https://api.github.com/users/mreiferson","html_url":"https://github.com/mreiferson","followers_url":"https://api.github.com/users/mreiferson/followers","following_url":"https://api.github.com/users/mreiferson/following{/other_user}","gists_url":"https://api.github.com/users/mreiferson/gists{/gist_id}","starred_url":"https://api.github.com/users/mreiferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreiferson/subscriptions","organizations_url":"https://api.github.com/users/mreiferson/orgs","repos_url":"https://api.github.com/users/mreiferson/repos","events_url":"https://api.github.com/users/mreiferson/events{/privacy}","received_events_url":"https://api.github.com/users/mreiferson/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mreiferson/nsq","description":"A realtime distributed messaging platform","fork":true,"url":"https://api.github.com/repos/mreiferson/nsq","forks_url":"https://api.github.com/repos/mreiferson/nsq/forks","keys_url":"https://api.github.com/repos/mreiferson/nsq/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mreiferson/nsq/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mreiferson/nsq/teams","hooks_url":"https://api.github.com/repos/mreiferson/nsq/hooks","issue_events_url":"https://api.github.com/repos/mreiferson/nsq/issues/events{/number}","events_url":"https://api.github.com/repos/mreiferson/nsq/events","assignees_url":"https://api.github.com/repos/mreiferson/nsq/assignees{/user}","branches_url":"https://api.github.com/repos/mreiferson/nsq/branches{/branch}","tags_url":"https://api.github.com/repos/mreiferson/nsq/tags","blobs_url":"https://api.github.com/repos/mreiferson/nsq/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mreiferson/nsq/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mreiferson/nsq/git/refs{/sha}","trees_url":"https://api.github.com/repos/mreiferson/nsq/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mreiferson/nsq/statuses/{sha}","languages_url":"https://api.github.com/repos/mreiferson/nsq/languages","stargazers_url":"https://api.github.com/repos/mreiferson/nsq/stargazers","contributors_url":"https://api.github.com/repos/mreiferson/nsq/contributors","subscribers_url":"https://api.github.com/repos/mreiferson/nsq/subscribers","subscription_url":"https://api.github.com/repos/mreiferson/nsq/subscription","commits_url":"https://api.github.com/repos/mreiferson/nsq/commits{/sha}","git_commits_url":"https://api.github.com/repos/mreiferson/nsq/git/commits{/sha}","comments_url":"https://api.github.com/repos/mreiferson/nsq/comments{/number}","issue_comment_url":"https://api.github.com/repos/mreiferson/nsq/issues/comments{/number}","contents_url":"https://api.github.com/repos/mreiferson/nsq/contents/{+path}","compare_url":"https://api.github.com/repos/mreiferson/nsq/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mreiferson/nsq/merges","archive_url":"https://api.github.com/repos/mreiferson/nsq/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mreiferson/nsq/downloads","issues_url":"https://api.github.com/repos/mreiferson/nsq/issues{/number}","pulls_url":"https://api.github.com/repos/mreiferson/nsq/pulls{/number}","milestones_url":"https://api.github.com/repos/mreiferson/nsq/milestones{/number}","notifications_url":"https://api.github.com/repos/mreiferson/nsq/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mreiferson/nsq/labels{/name}","releases_url":"https://api.github.com/repos/mreiferson/nsq/releases{/id}","created_at":"2012-05-16T04:03:26Z","updated_at":"2015-05-03T19:41:22Z","pushed_at":"2015-07-14T00:46:14Z","git_url":"git://github.com/mreiferson/nsq.git","ssh_url":"git@github.com:mreiferson/nsq.git","clone_url":"https://github.com/mreiferson/nsq.git","svn_url":"https://github.com/mreiferson/nsq","homepage":"http://bitly.github.io/nsq","size":8932,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":0,"forks":2,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"bitly:master","ref":"master","sha":"917a6c8c25e2f91ad8cd9696b89330b5bb799334","user":{"login":"bitly","id":251133,"avatar_url":"https://avatars.githubusercontent.com/u/251133?v=3","gravatar_id":"","url":"https://api.github.com/users/bitly","html_url":"https://github.com/bitly","followers_url":"https://api.github.com/users/bitly/followers","following_url":"https://api.github.com/users/bitly/following{/other_user}","gists_url":"https://api.github.com/users/bitly/gists{/gist_id}","starred_url":"https://api.github.com/users/bitly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bitly/subscriptions","organizations_url":"https://api.github.com/users/bitly/orgs","repos_url":"https://api.github.com/users/bitly/repos","events_url":"https://api.github.com/users/bitly/events{/privacy}","received_events_url":"https://api.github.com/users/bitly/received_events","type":"Organization","site_admin":false},"repo":{"id":4307108,"name":"nsq","full_name":"bitly/nsq","owner":{"login":"bitly","id":251133,"avatar_url":"https://avatars.githubusercontent.com/u/251133?v=3","gravatar_id":"","url":"https://api.github.com/users/bitly","html_url":"https://github.com/bitly","followers_url":"https://api.github.com/users/bitly/followers","following_url":"https://api.github.com/users/bitly/following{/other_user}","gists_url":"https://api.github.com/users/bitly/gists{/gist_id}","starred_url":"https://api.github.com/users/bitly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bitly/subscriptions","organizations_url":"https://api.github.com/users/bitly/orgs","repos_url":"https://api.github.com/users/bitly/repos","events_url":"https://api.github.com/users/bitly/events{/privacy}","received_events_url":"https://api.github.com/users/bitly/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bitly/nsq","description":"A realtime distributed messaging platform","fork":false,"url":"https://api.github.com/repos/bitly/nsq","forks_url":"https://api.github.com/repos/bitly/nsq/forks","keys_url":"https://api.github.com/repos/bitly/nsq/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bitly/nsq/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bitly/nsq/teams","hooks_url":"https://api.github.com/repos/bitly/nsq/hooks","issue_events_url":"https://api.github.com/repos/bitly/nsq/issues/events{/number}","events_url":"https://api.github.com/repos/bitly/nsq/events","assignees_url":"https://api.github.com/repos/bitly/nsq/assignees{/user}","branches_url":"https://api.github.com/repos/bitly/nsq/branches{/branch}","tags_url":"https://api.github.com/repos/bitly/nsq/tags","blobs_url":"https://api.github.com/repos/bitly/nsq/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bitly/nsq/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bitly/nsq/git/refs{/sha}","trees_url":"https://api.github.com/repos/bitly/nsq/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bitly/nsq/statuses/{sha}","languages_url":"https://api.github.com/repos/bitly/nsq/languages","stargazers_url":"https://api.github.com/repos/bitly/nsq/stargazers","contributors_url":"https://api.github.com/repos/bitly/nsq/contributors","subscribers_url":"https://api.github.com/repos/bitly/nsq/subscribers","subscription_url":"https://api.github.com/repos/bitly/nsq/subscription","commits_url":"https://api.github.com/repos/bitly/nsq/commits{/sha}","git_commits_url":"https://api.github.com/repos/bitly/nsq/git/commits{/sha}","comments_url":"https://api.github.com/repos/bitly/nsq/comments{/number}","issue_comment_url":"https://api.github.com/repos/bitly/nsq/issues/comments{/number}","contents_url":"https://api.github.com/repos/bitly/nsq/contents/{+path}","compare_url":"https://api.github.com/repos/bitly/nsq/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bitly/nsq/merges","archive_url":"https://api.github.com/repos/bitly/nsq/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bitly/nsq/downloads","issues_url":"https://api.github.com/repos/bitly/nsq/issues{/number}","pulls_url":"https://api.github.com/repos/bitly/nsq/pulls{/number}","milestones_url":"https://api.github.com/repos/bitly/nsq/milestones{/number}","notifications_url":"https://api.github.com/repos/bitly/nsq/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bitly/nsq/labels{/name}","releases_url":"https://api.github.com/repos/bitly/nsq/releases{/id}","created_at":"2012-05-12T14:37:08Z","updated_at":"2015-07-13T23:46:24Z","pushed_at":"2015-07-14T00:46:19Z","git_url":"git://github.com/bitly/nsq.git","ssh_url":"git@github.com:bitly/nsq.git","clone_url":"https://github.com/bitly/nsq.git","svn_url":"https://github.com/bitly/nsq","homepage":"http://nsq.io/","size":11948,"stargazers_count":5092,"watchers_count":5092,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":549,"mirror_url":null,"open_issues_count":38,"forks":549,"open_issues":38,"watchers":5092,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bitly/nsq/pulls/601"},"html":{"href":"https://github.com/bitly/nsq/pull/601"},"issue":{"href":"https://api.github.com/repos/bitly/nsq/issues/601"},"comments":{"href":"https://api.github.com/repos/bitly/nsq/issues/601/comments"},"review_comments":{"href":"https://api.github.com/repos/bitly/nsq/pulls/601/comments"},"review_comment":{"href":"https://api.github.com/repos/bitly/nsq/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bitly/nsq/pulls/601/commits"},"statuses":{"href":"https://api.github.com/repos/bitly/nsq/statuses/21ef3fe7ed46317ef266d7e0c95b28eae498cd53"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":7,"review_comments":0,"commits":8,"additions":483,"deletions":388,"changed_files":25}}