{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/1615","labels_url":"https://api.github.com/repos/druid-io/druid/issues/1615/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1615/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/1615/events","html_url":"https://github.com/druid-io/druid/issues/1615","id":100332389,"number":1615,"title":"[Proposal] : Better handling for *bad* worker nodes. ","user":{"login":"nishantmonu51","id":5023786,"avatar_url":"https://avatars.githubusercontent.com/u/5023786?v=3","gravatar_id":"","url":"https://api.github.com/users/nishantmonu51","html_url":"https://github.com/nishantmonu51","followers_url":"https://api.github.com/users/nishantmonu51/followers","following_url":"https://api.github.com/users/nishantmonu51/following{/other_user}","gists_url":"https://api.github.com/users/nishantmonu51/gists{/gist_id}","starred_url":"https://api.github.com/users/nishantmonu51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nishantmonu51/subscriptions","organizations_url":"https://api.github.com/users/nishantmonu51/orgs","repos_url":"https://api.github.com/users/nishantmonu51/repos","events_url":"https://api.github.com/users/nishantmonu51/events{/privacy}","received_events_url":"https://api.github.com/users/nishantmonu51/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/druid-io/druid/labels/Discuss","name":"Discuss","color":"ff00ff"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-08-11T14:58:13Z","updated_at":"2015-08-13T03:32:42Z","closed_at":null,"body":"Problem : \r\nIn case we have a bad worker having environment issues e.g disk issues and all tasks being assigned to that worker starts failing, If configured with FillCapacityWorkerSelectStrategy, Overlord keeps assigning new tasks to the bad worker since it is the most likely the one with the least used capacity. \r\n\r\n\r\nSolution :  \r\n\r\nApproach 1 : Add periodic sanity check to the workers.\r\n1) while starting up a worker will fork and run a sanity check task and announce itself if the task succeeds. If it fails it will retry it n (configurable) times and exits if the task fails repeatedly. \r\n2) worker will periodically run sanity checks on its own by forking a small test-task\r\n3) If the dummy task fails repeatedly, the worker disables itself and overlord will not assign any more tasks to this. \r\n\r\n Approach 2  : Make overlord smart to identify faulty workers and blacklist them. \r\n1) maintain a count of failed tasks, and blacklist a worker when it exceeds a configurable threshold. the counter will need to be reset when a task succeeds on a worker.\r\n2) send a small NoOp dummy task to the worker before blacklisting it in order to distinguish between issues in task itself or worker environment. \r\n3) Optionally also verify all the newly added workers too by assigning a dummy task. \r\n4) Autoscaling should be able to replace the blacklisted worker. \r\n\r\nI like Approach 1 and I think its better to have the worker do sanity checks on itself instead of overlord asking workers to do it. \r\n\r\nThoughts ? Suggestions? "},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/130519329","html_url":"https://github.com/druid-io/druid/issues/1615#issuecomment-130519329","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1615","id":130519329,"user":{"login":"himanshug","id":246308,"avatar_url":"https://avatars.githubusercontent.com/u/246308?v=3","gravatar_id":"","url":"https://api.github.com/users/himanshug","html_url":"https://github.com/himanshug","followers_url":"https://api.github.com/users/himanshug/followers","following_url":"https://api.github.com/users/himanshug/following{/other_user}","gists_url":"https://api.github.com/users/himanshug/gists{/gist_id}","starred_url":"https://api.github.com/users/himanshug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/himanshug/subscriptions","organizations_url":"https://api.github.com/users/himanshug/orgs","repos_url":"https://api.github.com/users/himanshug/repos","events_url":"https://api.github.com/users/himanshug/events{/privacy}","received_events_url":"https://api.github.com/users/himanshug/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T03:32:42Z","updated_at":"2015-08-13T03:32:42Z","body":"I like both. But, I would go for 1 because it is more self-contained within the worker and simpler to implement. Also, It will be great to increase the scope of 1 a little bit to track count of consecutive failures (for any tasks assigned to it and not just the sanity task, and disable itself if there were too many consecutive failures.\r\nIn future, We can evolve towards 2 if/when we notice cases not caught by 1."}}