{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5636","labels_url":"https://api.github.com/repos/apache/spark/issues/5636/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5636/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5636/events","html_url":"https://github.com/apache/spark/pull/5636","id":70197282,"number":5636,"title":"[SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException","user":{"login":"ilganeli","id":1483345,"avatar_url":"https://avatars.githubusercontent.com/u/1483345?v=3","gravatar_id":"","url":"https://api.github.com/users/ilganeli","html_url":"https://github.com/ilganeli","followers_url":"https://api.github.com/users/ilganeli/followers","following_url":"https://api.github.com/users/ilganeli/following{/other_user}","gists_url":"https://api.github.com/users/ilganeli/gists{/gist_id}","starred_url":"https://api.github.com/users/ilganeli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilganeli/subscriptions","organizations_url":"https://api.github.com/users/ilganeli/orgs","repos_url":"https://api.github.com/users/ilganeli/repos","events_url":"https://api.github.com/users/ilganeli/events{/privacy}","received_events_url":"https://api.github.com/users/ilganeli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":86,"created_at":"2015-04-22T18:14:08Z","updated_at":"2015-05-04T21:28:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5636","html_url":"https://github.com/apache/spark/pull/5636","diff_url":"https://github.com/apache/spark/pull/5636.diff","patch_url":"https://github.com/apache/spark/pull/5636.patch"},"body":"All - I've added a map to track the reasons, and counts / reason for stage failures. I then check whether to abort a stage based on whether there is a sufficient number of failures for a single reason.\r\n\r\nThe open questions are:\r\n\r\n1)  Is it really safe to assume that a ```FetchFailedException``` means that the ```BlockManager``` has failed, and it will work if we just try another one? E.g. does retrying the failed stage even make sense?\r\n2) For the equality check, I'm using the ```failureMessage``` inside the handler for ```FetchFailed```. Is it safe to assume that string will be consistent between subsequent stage failures or are there any counters or such in there that will change? If this isn't a safe item to use for comparison, is there any metric that can be used to track the number of failures for a particular reason?\r\n3) How many attempts for a stage should be made before aborting? I know that a task can have up to 4  failures before it's aborted but since a stage may see multiple fetch failures from concurrent tasks, I assume this number should be higher. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/98853959","html_url":"https://github.com/apache/spark/pull/5636#issuecomment-98853959","issue_url":"https://api.github.com/repos/apache/spark/issues/5636","id":98853959,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-05-04T21:28:51Z","updated_at":"2015-05-04T21:28:51Z","body":"  [Test build #31783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31783/consoleFull) for   PR 5636 at commit [`bc88aa1`](https://github.com/apache/spark/commit/bc88aa136021a6ae4bc1763005c357a718a32d89)."}}