{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4055","labels_url":"https://api.github.com/repos/apache/spark/issues/4055/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4055/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4055/events","html_url":"https://github.com/apache/spark/pull/4055","id":54410512,"number":4055,"title":"[SPARK-5259][CORE]Make sure shuffle metadata already in mapOutPutTracker while submitting tasks of the shuffleMapStage","user":{"login":"suyanNone","id":9818852,"avatar_url":"https://avatars.githubusercontent.com/u/9818852?v=3","gravatar_id":"","url":"https://api.github.com/users/suyanNone","html_url":"https://github.com/suyanNone","followers_url":"https://api.github.com/users/suyanNone/followers","following_url":"https://api.github.com/users/suyanNone/following{/other_user}","gists_url":"https://api.github.com/users/suyanNone/gists{/gist_id}","starred_url":"https://api.github.com/users/suyanNone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyanNone/subscriptions","organizations_url":"https://api.github.com/users/suyanNone/orgs","repos_url":"https://api.github.com/users/suyanNone/repos","events_url":"https://api.github.com/users/suyanNone/events{/privacy}","received_events_url":"https://api.github.com/users/suyanNone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":118,"created_at":"2015-01-15T04:07:57Z","updated_at":"2015-07-21T03:28:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4055","html_url":"https://github.com/apache/spark/pull/4055","diff_url":"https://github.com/apache/spark/pull/4055.diff","patch_url":"https://github.com/apache/spark/pull/4055.patch"},"body":"[SPARK-5259]Add task equal() and hashcode() to avoid stage.pendingTasks not accurate while stage was retry\r\ndesc:\r\nwhile run a spark job, it occurs one stage keep retrying and keep throwing FetchMetadataException\r\n\r\nreason:\r\nMap Stage 1-> Map Stage2\r\n\r\nMapStage1 retry, so have 2 taskSet, taskSet0.0 and TaskSet0.1\r\nTaskSet0.0 and TaskSet0.1 are all running.\r\n\r\nWhen to submit Map Stage2？\r\n```\r\n       if (!mapStage.isAvailable) {\r\n                  missing += mapStage\r\n                }\r\n\r\n  def isAvailable: Boolean = {\r\n    if (!isShuffleMap) {\r\n      true\r\n    } else {\r\n      numAvailableOutputs == numPartitions\r\n    }\r\n  }\r\n```\r\nhow numAvailableOutputs change?\r\n```\r\nstage.addOutputLoc(smt.partitionId, status)\r\n\r\n  def addOutputLoc(partition: Int, status: MapStatus) {\r\n    val prevList = outputLocs(partition)\r\n    outputLocs(partition) = status :: prevList\r\n    if (prevList == Nil) {\r\n      numAvailableOutputs += 1\r\n    }\r\n  }\r\n\r\n```\r\n\r\nWhen to register Map Stage1 out put?\r\n```\r\nif (runningStages.contains(stage) && stage.pendingTasks.isEmpty) {\r\n}\r\n```\r\nhow stage.pendingTasks change?\r\n```    \r\nevent.reason match {\r\n      case Success =>\r\n        listenerBus.post(SparkListenerTaskEnd(stageId, stage.latestInfo.attemptId, taskType,\r\n          event.reason, event.taskInfo, event.taskMetrics))\r\n        stage.pendingTasks -= task\r\n```\r\nbecause, because Task not override hashcode and equal, so run same partition task in different TaskSet is different task.  and pendingTask is clear when retry map stage, so the pendingTask is always for the new retry TaskSet. then the previous taskset complete some task which have same partition in the latest taskSet. `stage.pengdingTask -= task` not affect anything. but it affect `stage.numAvailableOutputs`,  because it just identified by partition Id. \r\nSo it may result in some stage have submit while his dependency map stage have not registered its output in MapOutputTracker. \r\n\r\n\r\n \r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/123140597","html_url":"https://github.com/apache/spark/pull/4055#issuecomment-123140597","issue_url":"https://api.github.com/repos/apache/spark/issues/4055","id":123140597,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T03:28:57Z","updated_at":"2015-07-21T03:28:57Z","body":"  [Test build #37905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37905/consoleFull) for   PR 4055 at commit [`83bba2b`](https://github.com/apache/spark/commit/83bba2b0a1dd3961eb534ca034c183598e5d32d6)."}}