{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10277","labels_url":"https://api.github.com/repos/apache/spark/issues/10277/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10277/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10277/events","html_url":"https://github.com/apache/spark/pull/10277","id":121860682,"number":10277,"title":"[SPARK-4621][shuffle]Shuffle index can be cached for SortShuffleManager in ExternalShuffle in order to  reduce indexFile's io","user":{"login":"lianhuiwang","id":545478,"avatar_url":"https://avatars.githubusercontent.com/u/545478?v=3","gravatar_id":"","url":"https://api.github.com/users/lianhuiwang","html_url":"https://github.com/lianhuiwang","followers_url":"https://api.github.com/users/lianhuiwang/followers","following_url":"https://api.github.com/users/lianhuiwang/following{/other_user}","gists_url":"https://api.github.com/users/lianhuiwang/gists{/gist_id}","starred_url":"https://api.github.com/users/lianhuiwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lianhuiwang/subscriptions","organizations_url":"https://api.github.com/users/lianhuiwang/orgs","repos_url":"https://api.github.com/users/lianhuiwang/repos","events_url":"https://api.github.com/users/lianhuiwang/events{/privacy}","received_events_url":"https://api.github.com/users/lianhuiwang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-12-12T15:04:43Z","updated_at":"2015-12-16T09:23:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10277","html_url":"https://github.com/apache/spark/pull/10277","diff_url":"https://github.com/apache/spark/pull/10277.diff","patch_url":"https://github.com/apache/spark/pull/10277.patch"},"body":"In ExternalShuffle shuffle index can cached for SortShuffleManager at first. Next time another reduce's fetching can read index from cache. Because reduce tasks fetch map output from ExternalShuffle at same time, shuffle index cache is effective.\r\nIn my test that it has 1000 partitions and 100 map tasks, it spends 3882ms for no cache index and it spends 608ms for cache index. So it is six times faster than index file.\r\n@rxin @JoshRosen @andrewor14 "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/165044305","html_url":"https://github.com/apache/spark/pull/10277#issuecomment-165044305","issue_url":"https://api.github.com/repos/apache/spark/issues/10277","id":165044305,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-12-16T09:23:02Z","updated_at":"2015-12-16T09:23:02Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47804/\nTest FAILed."}}