{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3798","labels_url":"https://api.github.com/repos/apache/spark/issues/3798/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3798/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3798/events","html_url":"https://github.com/apache/spark/pull/3798","id":52852469,"number":3798,"title":"[SPARK-4964] [Streaming] Exactly-once semantics for Kafka","user":{"login":"koeninger","id":98311,"avatar_url":"https://avatars.githubusercontent.com/u/98311?v=3","gravatar_id":"","url":"https://api.github.com/users/koeninger","html_url":"https://github.com/koeninger","followers_url":"https://api.github.com/users/koeninger/followers","following_url":"https://api.github.com/users/koeninger/following{/other_user}","gists_url":"https://api.github.com/users/koeninger/gists{/gist_id}","starred_url":"https://api.github.com/users/koeninger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koeninger/subscriptions","organizations_url":"https://api.github.com/users/koeninger/orgs","repos_url":"https://api.github.com/users/koeninger/repos","events_url":"https://api.github.com/users/koeninger/events{/privacy}","received_events_url":"https://api.github.com/users/koeninger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":48,"created_at":"2014-12-25T07:41:37Z","updated_at":"2015-01-29T02:49:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3798","html_url":"https://github.com/apache/spark/pull/3798","diff_url":"https://github.com/apache/spark/pull/3798.diff","patch_url":"https://github.com/apache/spark/pull/3798.patch"},"body":""},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/71960076","html_url":"https://github.com/apache/spark/pull/3798#issuecomment-71960076","issue_url":"https://api.github.com/repos/apache/spark/issues/3798","id":71960076,"user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"created_at":"2015-01-29T02:49:00Z","updated_at":"2015-01-29T02:49:00Z","body":"At a high level these are the high level design issues that are still pending \r\n\r\n- Packaging - I think all the classes should be org.apache.spark.streaming.kafka (even KafkaRDD), because (i) all of them are published in spark-streaming-kafka artifact, (ii) just importing one path (o.a.spark.streaming.kafka.KafkaUtils._) is sufficient get all the relevant classes. \r\n- KafkaUtils method name and scala doc - Lets keep thinking of names, but lets not stay blocked on this and continue focussing on other issues. \r\n- OffsetRange - I have proposed a design as well. Either we will go for Java interface or the simple class that I proposed. \r\n- Unit tests - No unit test for the new stream. Also is it possible to include a few unit tests for KafkaCluster?\r\n\r\nOther than these, I will start commenting on the code styles, etc very soon. \r\n\r\n\r\n"}}