{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8089","labels_url":"https://api.github.com/repos/apache/spark/issues/8089/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8089/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8089/events","html_url":"https://github.com/apache/spark/pull/8089","id":100204015,"number":8089,"title":"[SPARK-9808] Remove spark.shuffle.consolidateFiles and associated implementation.","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-08-11T01:35:04Z","updated_at":"2015-08-11T01:47:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8089","html_url":"https://github.com/apache/spark/pull/8089","diff_url":"https://github.com/apache/spark/pull/8089.diff","patch_url":"https://github.com/apache/spark/pull/8089.patch"},"body":"I think that we should remove `spark.shuffle.consolidateFiles` and its associated implementation for Spark 1.5.0. Rationale:\r\n\r\n- This feature is not properly tested. The sole test for this feature in `HashShuffleManagerSuite` does not appear to be testing the right thing because it never enables shuffle file consolidation.\r\n- The motivation for this feature, reducing the number of shuffle files created on disk, has been addressed by sort-based shuffle, which is better tested.\r\n- Shuffle file consolidation does not work with the external shuffle service."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/129670027","html_url":"https://github.com/apache/spark/pull/8089#issuecomment-129670027","issue_url":"https://api.github.com/repos/apache/spark/issues/8089","id":129670027,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T01:47:47Z","updated_at":"2015-08-11T01:47:47Z","body":" Merged build triggered."}}