{"action":"closed","issue":{"url":"https://api.github.com/repos/yasp-dota/yasp/issues/653","labels_url":"https://api.github.com/repos/yasp-dota/yasp/issues/653/labels{/name}","comments_url":"https://api.github.com/repos/yasp-dota/yasp/issues/653/comments","events_url":"https://api.github.com/repos/yasp-dota/yasp/issues/653/events","html_url":"https://github.com/yasp-dota/yasp/issues/653","id":106706892,"number":653,"title":"dynamic service discovery","user":{"login":"howardchung","id":3134520,"avatar_url":"https://avatars.githubusercontent.com/u/3134520?v=3","gravatar_id":"","url":"https://api.github.com/users/howardchung","html_url":"https://github.com/howardchung","followers_url":"https://api.github.com/users/howardchung/followers","following_url":"https://api.github.com/users/howardchung/following{/other_user}","gists_url":"https://api.github.com/users/howardchung/gists{/gist_id}","starred_url":"https://api.github.com/users/howardchung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/howardchung/subscriptions","organizations_url":"https://api.github.com/users/howardchung/orgs","repos_url":"https://api.github.com/users/howardchung/repos","events_url":"https://api.github.com/users/howardchung/events{/privacy}","received_events_url":"https://api.github.com/users/howardchung/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-16T06:16:20Z","updated_at":"2015-10-02T01:10:53Z","closed_at":"2015-10-02T01:10:53Z","body":"Optimally, yasp-core would handle parsers/retrievers dynamically joining/leaving the cluster.\r\n\r\nWe currently list all of the possible addresses of parsers/retrievers in our config.\r\n\r\nThis means:\r\n* Joining retrievers--we have to add them to config\r\n* Leaving retrievers--leads to delayed getReplayUrl since we may have to retry a few times before we get a valid retriever.\r\n* Joining parsers--we have to add them to config\r\n* Leaving parsers--The processors using that url will keep retrying and be stuck until the parser comes back online.\r\n\r\nI think the problem of dynamic service discovery is probably best solved by using something like https://www.consul.io/\r\nhttps://github.com/silas/node-consul\r\n\r\nIt means we have to install another dependency on yasp-core.  I think we only need to install node-consul on the service nodes, who register themselves with the yasp-core consul server.  It also doesn't solve the problem of not being able to dynamically change parse concurrency on-the-fly.\r\n\r\nSimpler service discovery:\r\nPerhaps we can simply set keys in redis with a ttl, and periodically refresh them by hitting the url without retry.  If they fail they simply expire from redis, and if they succeed they keep being found.\r\n\r\nStill doesn't solve the parallelism problem or the joining nodes problem, but solves the retrievers leaving problem."}}