{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8927","labels_url":"https://api.github.com/repos/apache/spark/issues/8927/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8927/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8927/events","html_url":"https://github.com/apache/spark/pull/8927","id":108582180,"number":8927,"title":"[SPARK-10796][CORE]Resubmit stage while lost task in Zombie TaskSets","user":{"login":"suyanNone","id":9818852,"avatar_url":"https://avatars.githubusercontent.com/u/9818852?v=3","gravatar_id":"","url":"https://api.github.com/users/suyanNone","html_url":"https://github.com/suyanNone","followers_url":"https://api.github.com/users/suyanNone/followers","following_url":"https://api.github.com/users/suyanNone/following{/other_user}","gists_url":"https://api.github.com/users/suyanNone/gists{/gist_id}","starred_url":"https://api.github.com/users/suyanNone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyanNone/subscriptions","organizations_url":"https://api.github.com/users/suyanNone/orgs","repos_url":"https://api.github.com/users/suyanNone/repos","events_url":"https://api.github.com/users/suyanNone/events{/privacy}","received_events_url":"https://api.github.com/users/suyanNone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-28T03:23:29Z","updated_at":"2015-09-28T10:02:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8927","html_url":"https://github.com/apache/spark/pull/8927","diff_url":"https://github.com/apache/spark/pull/8927.diff","patch_url":"https://github.com/apache/spark/pull/8927.patch"},"body":"We meet that problem in Spark 1.3.0, and I also check the latest Spark code, and I think that problem still exist.\r\ndesc:\r\n1. We know a running `ShuffleMapStage` will have multiple `TaskSet`: one Active TaskSet, multiple Zombie TaskSet. \r\n2. We think a running `ShuffleMapStage` is success only if its partition are all process success, namely each task‘s MapStatus are all add into `outputLocs`\r\n3. MapStatus of running `ShuffleMapStage` may succeed by Zombie TaskSet1 / Zombie TaskSet2 /..../ Active TaskSetN, and may some MapStatus only belong to one TaskSet, and may be a Zombie TaskSet.\r\n4. If lost a executor, it chanced that some lost-executor related MapStatus are succeed by some Zombie TaskSet. In current logical, The solution to resolved that lost MapStatus problem is, each TaskSet re-running that those tasks which succeed in lost-executor: re-add into `TaskSet's pendingTasks`, and re-add it paritions into `Stage‘s pendingPartitions` . but it is useless if that lost MapStatus only belong to Zombie TaskSet, it is Zombie, so will never be scheduled his `pendingTasks`\r\n5. The condition for resubmit stage is only if some task throws `FetchFailedException`, but may the lost-executor just not empty any MapStatus of parent Stage for one of running Stages, and it‘s happen to that running `Stage` was lost a MapStatus only belong to a ZombieTask. So if all Zombie TaskSets are all processed his runningTasks and Active TaskSet are all processed his pendingTask, then will removed by `TaskSchedulerImp`, then that running Stage's pending partitions is still nonEmpty. it will hangs......\r\n\r\n\r\nExamples:\r\nRunning Stage 0.0, running TaskSet0.0, Finshed task0.0 in ExecA, running Task1.0 in ExecB, waiting Task2.0\r\n---> Task1.0 throws FetchFailedException\r\n---> Running Resubmited stage 0.1, running TaskSet0.1(which re-run Task1, Task2), assume Task 1.0 finshed in ExecA\r\n---> ExecA lost, and it happens no one throw FetchFailedExecption.\r\n---> TaskSet0.1 re-submit task 1, re-add it into pendingTasks, and waiting TaskSchedulerImp schedule.\r\nTaskSet 0.0 also resubmit task0, re-add it into pendingTasks, because it‘s Zombie, TaskSchedulerImpl skip to schedule TaskSet0.0\r\nSo if TaskSet0.0 and TaskSet0.1 (isZombie && runningTasks.empty), TaskSchedulerImp will remove those TaskSets.\r\nDagScheduler still have pendingPartitions due to the task lost in TaskSet0.0, but his TaskSets are all removed, so hangs....\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/143698641","html_url":"https://github.com/apache/spark/pull/8927#issuecomment-143698641","issue_url":"https://api.github.com/repos/apache/spark/issues/8927","id":143698641,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-09-28T10:02:13Z","updated_at":"2015-09-28T10:02:13Z","body":" Merged build triggered."}}