{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4550","labels_url":"https://api.github.com/repos/apache/spark/issues/4550/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4550/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4550/events","html_url":"https://github.com/apache/spark/pull/4550","id":57411649,"number":4550,"title":"[SPARK-3570] Include time to open files in shuffle write time.","user":{"login":"kayousterhout","id":1108612,"avatar_url":"https://avatars.githubusercontent.com/u/1108612?v=3","gravatar_id":"","url":"https://api.github.com/users/kayousterhout","html_url":"https://github.com/kayousterhout","followers_url":"https://api.github.com/users/kayousterhout/followers","following_url":"https://api.github.com/users/kayousterhout/following{/other_user}","gists_url":"https://api.github.com/users/kayousterhout/gists{/gist_id}","starred_url":"https://api.github.com/users/kayousterhout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayousterhout/subscriptions","organizations_url":"https://api.github.com/users/kayousterhout/orgs","repos_url":"https://api.github.com/users/kayousterhout/repos","events_url":"https://api.github.com/users/kayousterhout/events{/privacy}","received_events_url":"https://api.github.com/users/kayousterhout/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-02-12T03:10:41Z","updated_at":"2015-02-14T00:44:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4550","html_url":"https://github.com/apache/spark/pull/4550","diff_url":"https://github.com/apache/spark/pull/4550.diff","patch_url":"https://github.com/apache/spark/pull/4550.patch"},"body":"Opening shuffle files can be very significant when the disk is\r\ncontended, especially when using ext3. While writing data to\r\na file can avoid hitting disk (and instead hit the buffer\r\ncache), opening a file always involves writing some metadata\r\nabout the file to disk, so the open time can be a very significant\r\nportion of the shuffle write time. In one job I ran recently, the time to\r\nwrite shuffle data to the file was only 4ms for each task, but\r\nthe time to open the file was about 100x as long (~400ms).\r\n\r\nWhen we add metrics about spilled data (#2504), we should ensure\r\nthat the file open time is also included there."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/74352107","html_url":"https://github.com/apache/spark/pull/4550#issuecomment-74352107","issue_url":"https://api.github.com/repos/apache/spark/issues/4550","id":74352107,"user":{"login":"sryza","id":654855,"avatar_url":"https://avatars.githubusercontent.com/u/654855?v=3","gravatar_id":"","url":"https://api.github.com/users/sryza","html_url":"https://github.com/sryza","followers_url":"https://api.github.com/users/sryza/followers","following_url":"https://api.github.com/users/sryza/following{/other_user}","gists_url":"https://api.github.com/users/sryza/gists{/gist_id}","starred_url":"https://api.github.com/users/sryza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sryza/subscriptions","organizations_url":"https://api.github.com/users/sryza/orgs","repos_url":"https://api.github.com/users/sryza/repos","events_url":"https://api.github.com/users/sryza/events{/privacy}","received_events_url":"https://api.github.com/users/sryza/received_events","type":"User","site_admin":false},"created_at":"2015-02-14T00:44:07Z","updated_at":"2015-02-14T00:44:07Z","body":"Ah I see.  Is it possible that we might hit higher latencies for the main path when there's high contention (more cores than disks)?\r\n\r\nIf that seems unlikely, sounds fine to me to just add a comment.\r\n"}}