{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6883","labels_url":"https://api.github.com/repos/apache/spark/issues/6883/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6883/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6883/events","html_url":"https://github.com/apache/spark/pull/6883","id":89311722,"number":6883,"title":"SPARK-4644 blockjoin","user":{"login":"koertkuipers","id":821284,"avatar_url":"https://avatars.githubusercontent.com/u/821284?v=3","gravatar_id":"","url":"https://api.github.com/users/koertkuipers","html_url":"https://github.com/koertkuipers","followers_url":"https://api.github.com/users/koertkuipers/followers","following_url":"https://api.github.com/users/koertkuipers/following{/other_user}","gists_url":"https://api.github.com/users/koertkuipers/gists{/gist_id}","starred_url":"https://api.github.com/users/koertkuipers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koertkuipers/subscriptions","organizations_url":"https://api.github.com/users/koertkuipers/orgs","repos_url":"https://api.github.com/users/koertkuipers/repos","events_url":"https://api.github.com/users/koertkuipers/events{/privacy}","received_events_url":"https://api.github.com/users/koertkuipers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-06-18T14:41:54Z","updated_at":"2015-06-19T20:17:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6883","html_url":"https://github.com/apache/spark/pull/6883","diff_url":"https://github.com/apache/spark/pull/6883.diff","patch_url":"https://github.com/apache/spark/pull/6883.patch"},"body":"Although the discussion (and design doc) under SPARK-4644 seem focussed on other aspects of skew (OOM mostly) than this pullreq (which focusses on avoiding a single reducer taking a long time), i decided to put this pullreq under SPARK-4644 anyhow, to avoid the proliferation of JIRA tickets. If this is not the right place let me know and i will move it.\r\n\r\nInspired by block join in scalding.\r\nFrom scalding docs:\r\n\r\nThis is useful in cases where the data has extreme skew. A symptom of this is that we may see a job stuck for a very long time on a small number of reducers.\r\nA block join is way to get around this: we add a random integer field and a replica field to every tuple in the left and right pipes. We then join on the original keys and on these new dummy fields. These dummy fields make it less likely that the skewed keys will be hashed to the same reducer.\r\n\r\nThe final data size is right * rightReplication + left * leftReplication but because of the fragmentation, we are guaranteed the same number of hits as the original join.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/113630477","html_url":"https://github.com/apache/spark/pull/6883#issuecomment-113630477","issue_url":"https://api.github.com/repos/apache/spark/issues/6883","id":113630477,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-06-19T20:17:33Z","updated_at":"2015-06-19T20:17:33Z","body":"Merged build finished. Test FAILed."}}