{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5385","labels_url":"https://api.github.com/repos/apache/spark/issues/5385/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5385/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5385/events","html_url":"https://github.com/apache/spark/pull/5385","id":66794404,"number":5385,"title":"[SPARK-6691][Streaming][WIP] Add a dynamic RateLimiter for Spark Streaming","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-04-07T06:36:40Z","updated_at":"2015-06-19T01:52:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5385","html_url":"https://github.com/apache/spark/pull/5385","diff_url":"https://github.com/apache/spark/pull/5385.diff","patch_url":"https://github.com/apache/spark/pull/5385.patch"},"body":"This proposal add a dynamic RateLimiter for Spark Streaming's both receiver and direct based input streams. The details can be seen in [SPARK-6691](https://issues.apache.org/jira/browse/SPARK-6691)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/113338415","html_url":"https://github.com/apache/spark/pull/5385#issuecomment-113338415","issue_url":"https://api.github.com/repos/apache/spark/issues/5385","id":113338415,"user":{"login":"harishreedharan","id":1481194,"avatar_url":"https://avatars.githubusercontent.com/u/1481194?v=3","gravatar_id":"","url":"https://api.github.com/users/harishreedharan","html_url":"https://github.com/harishreedharan","followers_url":"https://api.github.com/users/harishreedharan/followers","following_url":"https://api.github.com/users/harishreedharan/following{/other_user}","gists_url":"https://api.github.com/users/harishreedharan/gists{/gist_id}","starred_url":"https://api.github.com/users/harishreedharan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harishreedharan/subscriptions","organizations_url":"https://api.github.com/users/harishreedharan/orgs","repos_url":"https://api.github.com/users/harishreedharan/repos","events_url":"https://api.github.com/users/harishreedharan/events{/privacy}","received_events_url":"https://api.github.com/users/harishreedharan/received_events","type":"User","site_admin":false},"created_at":"2015-06-19T01:52:12Z","updated_at":"2015-06-19T01:52:12Z","body":"On Thursday, June 18, 2015, andrewor14 <notifications@github.com> wrote:\n\n> Well, there are at least two other important considerations:\n>\n>    - We should use the batch queue instead of the task queue\n>\n> Not sure what you mean here - can you explain a bit? In the receiver case\npartition count depends on amount of data received while in the diret\nstream case it is fixed, so that we'd have to rethink since the task count\nis constant.\n\n>\n>    - We should never kill executors with receivers\n>\n> This is not an issue since the receiver is basically executed as a long\nrunning task.\n\n>\n>\n> The first is important because dynamic allocation currently doesn't really\n> do anything in most streaming workloads. The second is crucial because we\n> don't want dynamic allocation to disrupt the basic function of the\n> application.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/apache/spark/pull/5385#issuecomment-113338014>.\n>\n\n\n-- \n\nThanks,\nHari\n"}}