{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4964","labels_url":"https://api.github.com/repos/apache/spark/issues/4964/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4964/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4964/events","html_url":"https://github.com/apache/spark/pull/4964","id":60564248,"number":4964,"title":"[SPARK-6222][STREAMING] Checkpoints should be done only after batches ar...","user":{"login":"harishreedharan","id":1481194,"avatar_url":"https://avatars.githubusercontent.com/u/1481194?v=3","gravatar_id":"","url":"https://api.github.com/users/harishreedharan","html_url":"https://github.com/harishreedharan","followers_url":"https://api.github.com/users/harishreedharan/followers","following_url":"https://api.github.com/users/harishreedharan/following{/other_user}","gists_url":"https://api.github.com/users/harishreedharan/gists{/gist_id}","starred_url":"https://api.github.com/users/harishreedharan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harishreedharan/subscriptions","organizations_url":"https://api.github.com/users/harishreedharan/orgs","repos_url":"https://api.github.com/users/harishreedharan/repos","events_url":"https://api.github.com/users/harishreedharan/events{/privacy}","received_events_url":"https://api.github.com/users/harishreedharan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-10T20:29:25Z","updated_at":"2015-03-10T21:51:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4964","html_url":"https://github.com/apache/spark/pull/4964","diff_url":"https://github.com/apache/spark/pull/4964.diff","patch_url":"https://github.com/apache/spark/pull/4964.patch"},"body":"...e completed.\r\n\r\nCurrently checkpoints are initiated as soon as a batch is submitted for processing. We\r\ndo not wait for the processing to complete before starting a checkpoint.\r\n\r\nThe checkpoint at a given time actually does not reflect what has been processed,\r\nso even for Dstreams that do not use the WAL data loss is possible. For example, the direct\r\nkafka connector, the checkpoint has a range of offsets that have not been processed, leading\r\nto data loss.\r\n\r\nIn addition, the WAL files which are deleted at the end of a checkpoint may\r\ncontain unprocessed data, if the number of batches waiting to be processed is more\r\nthan a handful - and can result in data loss."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/78156435","html_url":"https://github.com/apache/spark/pull/4964#issuecomment-78156435","issue_url":"https://api.github.com/repos/apache/spark/issues/4964","id":78156435,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-03-10T21:51:53Z","updated_at":"2015-03-10T21:51:53Z","body":"  [Test build #28439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28439/consoleFull) for   PR 4964 at commit [`076b134`](https://github.com/apache/spark/commit/076b1344776f4e0d0242217aa0c0a823727994ab).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}