{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5852","labels_url":"https://api.github.com/repos/apache/spark/issues/5852/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5852/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5852/events","html_url":"https://github.com/apache/spark/pull/5852","id":72541578,"number":5852,"title":"[SPARK-6166] [Spark Core] Limit number of concurrent outbound connections.","user":{"login":"mridulm","id":1591700,"avatar_url":"https://avatars.githubusercontent.com/u/1591700?v=3","gravatar_id":"","url":"https://api.github.com/users/mridulm","html_url":"https://github.com/mridulm","followers_url":"https://api.github.com/users/mridulm/followers","following_url":"https://api.github.com/users/mridulm/following{/other_user}","gists_url":"https://api.github.com/users/mridulm/gists{/gist_id}","starred_url":"https://api.github.com/users/mridulm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mridulm/subscriptions","organizations_url":"https://api.github.com/users/mridulm/orgs","repos_url":"https://api.github.com/users/mridulm/repos","events_url":"https://api.github.com/users/mridulm/events{/privacy}","received_events_url":"https://api.github.com/users/mridulm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-05-01T22:47:08Z","updated_at":"2015-05-02T00:27:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5852","html_url":"https://github.com/apache/spark/pull/5852","diff_url":"https://github.com/apache/spark/pull/5852.diff","patch_url":"https://github.com/apache/spark/pull/5852.patch"},"body":"\r\nThe overall number of requests in the cluster comes down with this (though in theory, you can still have one node getting swamped). In our experiments this has allowed jobs to progress where they previously die due to workers repeatedly going down when number of requests go up.\r\nTypically happens when you have large number of nodes, and data per node is not too high (in one of the many jobs in the application) - drastic lowering of spark.reducer.maxSizeInFlightMb results is very poor performance overall.\r\n\r\n\r\nNote: I have not included any testcases - suggestions on how to do the same in current spark test framework would be great - I have not looked at spark test suites in a while. Simulating this is slightly 'interesting'"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/98268260","html_url":"https://github.com/apache/spark/pull/5852#issuecomment-98268260","issue_url":"https://api.github.com/repos/apache/spark/issues/5852","id":98268260,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-05-02T00:27:06Z","updated_at":"2015-05-02T00:27:06Z","body":"  [Test build #31623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31623/consoleFull) for   PR 5852 at commit [`32168fc`](https://github.com/apache/spark/commit/32168fc7a14cc341220ebf7548f568acc7e774d0).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}