{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8687","labels_url":"https://api.github.com/repos/apache/spark/issues/8687/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8687/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8687/events","html_url":"https://github.com/apache/spark/pull/8687","id":105747180,"number":8687,"title":"[SPARK-9899][SQL] log warning for direct output committer with speculation enabled","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-09-10T07:31:01Z","updated_at":"2015-09-12T07:12:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8687","html_url":"https://github.com/apache/spark/pull/8687","diff_url":"https://github.com/apache/spark/pull/8687.diff","patch_url":"https://github.com/apache/spark/pull/8687.patch"},"body":"This is a follow-up of https://github.com/apache/spark/pull/8317.\r\n\r\nWhen speculation is enabled, there may be multiply tasks writing to the same path. Generally it's OK as we will write to a temporary directory first and only one task can commit the temporary directory to target path.\r\n\r\nHowever, when we use direct output committer, tasks will write data to target path directly without temporary directory. This causes problems like corrupted data. Please see [PR comment](https://github.com/apache/spark/pull/8191#issuecomment-131598385) for more details.\r\n\r\nUnfortunately, we don't have a simple flag to tell if a output committer will write to temporary directory or not, so for safety, we have to disable any customized output committer when `speculation` is true."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/139737781","html_url":"https://github.com/apache/spark/pull/8687#issuecomment-139737781","issue_url":"https://api.github.com/repos/apache/spark/issues/8687","id":139737781,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-09-12T07:12:13Z","updated_at":"2015-09-12T07:12:13Z","body":" Merged build triggered."}}