{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4155","labels_url":"https://api.github.com/repos/apache/spark/issues/4155/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4155/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4155/events","html_url":"https://github.com/apache/spark/pull/4155","id":55105184,"number":4155,"title":"[SPARK-4879] Use the Spark driver to authorize Hadoop commits.","user":{"login":"mccheah","id":8975385,"avatar_url":"https://avatars.githubusercontent.com/u/8975385?v=3","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2015-01-22T02:01:55Z","updated_at":"2015-01-23T03:37:49Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4155","html_url":"https://github.com/apache/spark/pull/4155","diff_url":"https://github.com/apache/spark/pull/4155.diff","patch_url":"https://github.com/apache/spark/pull/4155.patch"},"body":"This is a version of https://github.com/apache/spark/pull/4066/ which is up to date with master and has unit tests.\r\n\r\nPreviously, SparkHadoopWriter always committed its tasks without question. The problem is that when speculation is turned on, sometimes this can result in multiple tasks committing their output to the same\r\npartition. Even though an HDFS-writing task may be re-launched due to speculation, the original task is not killed and may eventually commit as well.\r\n\r\nThis can cause strange race conditions where multiple tasks that commit interfere with each other, with the result being that some partition files are actually lost entirely. For more context on these kinds of scenarios, see the aforementioned JIRA ticket.\r\n\r\nIn Hadoop MapReduce jobs, the application master is a central coordinator that authorizes whether or not any given task can commit. Before a task commits its output, it queries the application master as to whether or not such a commit is safe, and the application master does bookkeeping as tasks are requesting commits. Duplicate tasks that would write to files that were already written to from other tasks are prohibited from committing.\r\n\r\nThis patch emulates that functionality - the crucial missing component was a central arbitrator, which is now a module called the OutputCommitCoordinator. The coordinator lives on the driver and the executors can obtain a reference to this actor and request its permission to commit. As tasks commit and are reported as completed successfully or unsuccessfully by the DAGScheduler, the commit coordinator is informed of the task completion events as well to update its internal state.\r\n\r\nFuture work includes more rigorous unit testing and extra optimizations should this patch cause a performance regression. It is unclear what the overall cost of communicating back to the driver on every hadoop-committing task will be. It's also important for those hitting this issue to backport this onto previous version of Spark because the bug has serious consequences, that is, data is lost.\r\n\r\ncc @MingyuKim @JoshRosen  @ash211 "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/71143090","html_url":"https://github.com/apache/spark/pull/4155#issuecomment-71143090","issue_url":"https://api.github.com/repos/apache/spark/issues/4155","id":71143090,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-01-23T03:37:49Z","updated_at":"2015-01-23T03:37:49Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25986/\nTest FAILed."}}