{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5572","labels_url":"https://api.github.com/repos/apache/spark/issues/5572/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5572/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5572/events","html_url":"https://github.com/apache/spark/pull/5572","id":69313785,"number":5572,"title":"[SPARK-6307][Core] Speed up RDD.cartesian by caching remotely received blocks","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":84,"created_at":"2015-04-18T14:29:01Z","updated_at":"2015-07-15T04:32:46Z","closed_at":"2015-07-15T04:32:46Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5572","html_url":"https://github.com/apache/spark/pull/5572","diff_url":"https://github.com/apache/spark/pull/5572.diff","patch_url":"https://github.com/apache/spark/pull/5572.patch"},"body":"JIRA: https://issues.apache.org/jira/browse/SPARK-6307 and https://issues.apache.org/jira/browse/SPARK-6922\r\n\r\nThe problematic cases are stated clearly in the related JIRAs.\r\n\r\nCurrently, `CacheManager.getOrCompute` only caches locally computed values. If a required block is retrieved remotely, it will not be cached. In most use cases, this is working well because the same block will not be required in multiple times.\r\n\r\nHowever, `CartesianRDD` hits this problem and it will require same blocks in multiple times to compute the cross product split. Thus, the same blocks are fetched again and again and the performance is very slow.\r\n\r\nThis pr tried to cache the remotely received blocks when performing cartesian product on rdds."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/121485194","html_url":"https://github.com/apache/spark/pull/5572#issuecomment-121485194","issue_url":"https://api.github.com/repos/apache/spark/issues/5572","id":121485194,"user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2015-07-15T04:32:46Z","updated_at":"2015-07-15T04:32:46Z","body":"Close this PR first as I don't know if it is needed now."}}