{"action":"created","issue":{"url":"https://api.github.com/repos/amplab/tachyon/issues/702","labels_url":"https://api.github.com/repos/amplab/tachyon/issues/702/labels{/name}","comments_url":"https://api.github.com/repos/amplab/tachyon/issues/702/comments","events_url":"https://api.github.com/repos/amplab/tachyon/issues/702/events","html_url":"https://github.com/amplab/tachyon/pull/702","id":60446894,"number":702,"title":"[TACHYON-295] Only attempt to read from worker memory once per block stream.","user":{"login":"calvinjia","id":2223813,"avatar_url":"https://avatars.githubusercontent.com/u/2223813?v=3","gravatar_id":"","url":"https://api.github.com/users/calvinjia","html_url":"https://github.com/calvinjia","followers_url":"https://api.github.com/users/calvinjia/followers","following_url":"https://api.github.com/users/calvinjia/following{/other_user}","gists_url":"https://api.github.com/users/calvinjia/gists{/gist_id}","starred_url":"https://api.github.com/users/calvinjia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calvinjia/subscriptions","organizations_url":"https://api.github.com/users/calvinjia/orgs","repos_url":"https://api.github.com/users/calvinjia/repos","events_url":"https://api.github.com/users/calvinjia/events{/privacy}","received_events_url":"https://api.github.com/users/calvinjia/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-03-10T03:42:26Z","updated_at":"2015-03-11T06:19:13Z","closed_at":"2015-03-11T06:07:59Z","pull_request":{"url":"https://api.github.com/repos/amplab/tachyon/pulls/702","html_url":"https://github.com/amplab/tachyon/pull/702","diff_url":"https://github.com/amplab/tachyon/pull/702.diff","patch_url":"https://github.com/amplab/tachyon/pull/702.patch"},"body":"Since multiple reads will most likely occur for each block, trying to find a copy of the block in a worker's memory is very expensive each time and unlikely to succeed after the first failure.\r\n\r\nThe performance improvement is around 240s -> 10s when reading a 1.5 gb file in HDFS on my local machine."},"comment":{"url":"https://api.github.com/repos/amplab/tachyon/issues/comments/78211327","html_url":"https://github.com/amplab/tachyon/pull/702#issuecomment-78211327","issue_url":"https://api.github.com/repos/amplab/tachyon/issues/702","id":78211327,"user":{"login":"carsonwang","id":9278199,"avatar_url":"https://avatars.githubusercontent.com/u/9278199?v=3","gravatar_id":"","url":"https://api.github.com/users/carsonwang","html_url":"https://github.com/carsonwang","followers_url":"https://api.github.com/users/carsonwang/followers","following_url":"https://api.github.com/users/carsonwang/following{/other_user}","gists_url":"https://api.github.com/users/carsonwang/gists{/gist_id}","starred_url":"https://api.github.com/users/carsonwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/carsonwang/subscriptions","organizations_url":"https://api.github.com/users/carsonwang/orgs","repos_url":"https://api.github.com/users/carsonwang/repos","events_url":"https://api.github.com/users/carsonwang/events{/privacy}","received_events_url":"https://api.github.com/users/carsonwang/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T06:19:13Z","updated_at":"2015-03-11T06:19:13Z","body":"In my test, it takes 16-17s for reading 1GB from a remote worker's memory. Most of the overhead is caused by network. If the tachyon client is on a datanode where all blocks of the HDFS file in on the node, it only takes 3.2 seconds reading 1GB data from HDFS. One a namenode, it takes about 10s reading from HDFS which is still faster than reading from a remote worker's memory."}}