{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/884","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/884/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/884/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/884/events","html_url":"https://github.com/Yelp/mrjob/issues/884","id":28737804,"number":884,"title":"Raise error and abort job if certain percentage of task trackers are blacklisted","user":{"login":"sudarshang","id":119499,"avatar_url":"https://avatars.githubusercontent.com/u/119499?v=3","gravatar_id":"","url":"https://api.github.com/users/sudarshang","html_url":"https://github.com/sudarshang","followers_url":"https://api.github.com/users/sudarshang/followers","following_url":"https://api.github.com/users/sudarshang/following{/other_user}","gists_url":"https://api.github.com/users/sudarshang/gists{/gist_id}","starred_url":"https://api.github.com/users/sudarshang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sudarshang/subscriptions","organizations_url":"https://api.github.com/users/sudarshang/orgs","repos_url":"https://api.github.com/users/sudarshang/repos","events_url":"https://api.github.com/users/sudarshang/events{/privacy}","received_events_url":"https://api.github.com/users/sudarshang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-03-04T20:28:55Z","updated_at":"2015-03-11T00:51:36Z","closed_at":null,"body":"In case of long running jobs - (jobs running for a day or two) we notice that all the task trackers are blacklisted and the job stops making any progress. The AWS CloudWatch API provides a way to monitor the tack trackers. We should have an EMR specific options that allows the user to specify a certain percentage of task trackers that must be up otherwise the job will be aborted.\r\n"},"comment":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/78180321","html_url":"https://github.com/Yelp/mrjob/issues/884#issuecomment-78180321","issue_url":"https://api.github.com/repos/Yelp/mrjob/issues/884","id":78180321,"user":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T00:51:36Z","updated_at":"2015-03-11T00:51:36Z","body":"Last time I tried to use CloudWatch, I determined it couldn't actually terminate clusters (see #628).\r\n\r\nYou should be able to see the task trackers from the master node, yes? We could do something similar to the idle job termination script that does a health check on the task trackers."}}