{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5908","labels_url":"https://api.github.com/repos/apache/spark/issues/5908/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5908/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5908/events","html_url":"https://github.com/apache/spark/pull/5908","id":73268234,"number":5908,"title":"[SPARK-7389] [core]Tachyon integration improvement","user":{"login":"shimingfei","id":2638130,"avatar_url":"https://avatars.githubusercontent.com/u/2638130?v=3","gravatar_id":"","url":"https://api.github.com/users/shimingfei","html_url":"https://github.com/shimingfei","followers_url":"https://api.github.com/users/shimingfei/followers","following_url":"https://api.github.com/users/shimingfei/following{/other_user}","gists_url":"https://api.github.com/users/shimingfei/gists{/gist_id}","starred_url":"https://api.github.com/users/shimingfei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shimingfei/subscriptions","organizations_url":"https://api.github.com/users/shimingfei/orgs","repos_url":"https://api.github.com/users/shimingfei/repos","events_url":"https://api.github.com/users/shimingfei/events{/privacy}","received_events_url":"https://api.github.com/users/shimingfei/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":44,"created_at":"2015-05-05T10:06:33Z","updated_at":"2015-05-18T03:53:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5908","html_url":"https://github.com/apache/spark/pull/5908","diff_url":"https://github.com/apache/spark/pull/5908.diff","patch_url":"https://github.com/apache/spark/pull/5908.patch"},"body":"Two main changes:\r\n\r\nAdd two functions in ExternalBlockManager, which are putValues and getValues\r\nbecause the implementation may not rely on the putBytes and getBytes\r\n\r\nimprove Tachyon integration.\r\nCurrently, when putting data into Tachyon, Spark first serialize all data in one partition into a ByteBuffer, and then write into Tachyon, this will uses much memory and increase GC overhead\r\n\r\nwhen get data from Tachyon, getValues depends on getBytes, which also read all data into On heap byte arry, and result in much memory usage.\r\nThis PR changes the approach of the two functions, make them read / write data by stream to reduce memory usage.\r\n\r\nIn our testing,  when data size is huge, this patch reduces about 30% GC time and 70% full GC time, and total execution time reduces about 10% \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/102915182","html_url":"https://github.com/apache/spark/pull/5908#issuecomment-102915182","issue_url":"https://api.github.com/repos/apache/spark/issues/5908","id":102915182,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-05-18T03:53:09Z","updated_at":"2015-05-18T03:53:09Z","body":"Merged build finished. Test PASSed."}}