{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10197","labels_url":"https://api.github.com/repos/apache/spark/issues/10197/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10197/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10197/events","html_url":"https://github.com/apache/spark/pull/10197","id":120975059,"number":10197,"title":"[SPARK-12203][STREAMING] Add KafkaDirectInputDStream","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2015-12-08T10:06:12Z","updated_at":"2015-12-13T05:37:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10197","html_url":"https://github.com/apache/spark/pull/10197","diff_url":"https://github.com/apache/spark/pull/10197.diff","patch_url":"https://github.com/apache/spark/pull/10197.patch"},"body":"JIRA: https://issues.apache.org/jira/browse/SPARK-12203\r\n\r\nCurrently, we have DirectKafkaInputDStream, which directly pulls messages from Kafka Brokers without any receivers, and KafkaInputDStream, which pulls messages from a Kafka Broker using receivers with zookeeper.\r\n\r\nAs we observed, because DirectKafkaInputDStream retrieves messages from Kafka after each batch finishes, it posts a latency compared with KafkaInputDStream that continues to pull messages during each batch window.\r\n\r\nSo we try to add KafkaDirectInputDStream that directly pulls messages from Kafka Brokers as DirectKafkaInputDStream, but it uses receivers as KafkaInputDStream and pulls messages as blocks during each batch window.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/164228478","html_url":"https://github.com/apache/spark/pull/10197#issuecomment-164228478","issue_url":"https://api.github.com/repos/apache/spark/issues/10197","id":164228478,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-12-13T05:37:11Z","updated_at":"2015-12-13T05:37:11Z","body":"Merged build finished. Test PASSed."}}