{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/47552258","id":47552258,"diff_hunk":"@@ -715,6 +735,56 @@ class CheckpointSuite extends TestSuiteBase with DStreamCheckpointTester {\n     }\n   }\n \n+  test(\"DStream checkpoint invoke times\") {\n+    var clock: ManualClock = null","path":"streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala","position":46,"original_position":46,"commit_id":"ac8795197f6fa552b4215c63c93b6112652972d9","original_commit_id":"ac8795197f6fa552b4215c63c93b6112652972d9","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"body":"I suggest writing a simpler test that only tests this issue, such as:\r\n```\r\n    withStreamingContext(new StreamingContext(conf, batchDuration)) { ssc =>\r\n      ssc.checkpoint(checkpointDir)\r\n      val inputDStream = new CheckpointInputDStream(ssc)\r\n      val checkpointData = inputDStream.checkpointData\r\n      val mappedDStream = inputDStream.map(_ + 100)\r\n      val outputStream = new TestOutputStreamWithPartitions(mappedDStream)\r\n      outputStream.register()\r\n      /// do more two times output\r\n      mappedDStream.foreachRDD(rdd => rdd.count())\r\n      mappedDStream.foreachRDD(rdd => rdd.count())\r\n      assert(checkpointData.restoredTimes === 0)\r\n      val batchDurationMillis = ssc.progressListener.batchDuration\r\n      generateOutput(ssc, Time(batchDurationMillis * 3), checkpointDir, stopSparkContext = false)\r\n      assert(checkpointData.restoredTimes === 0)\r\n    }\r\n    withStreamingContext(new StreamingContext(checkpointDir)) { ssc =>\r\n      val checkpointData =\r\n        ssc.graph.getInputStreams().head.asInstanceOf[CheckpointInputDStream].checkpointData\r\n      ssc.start()\r\n      ssc.stop()\r\n      assert(checkpointData.restoredTimes === 1)\r\n    }\r\n```","created_at":"2015-12-14T20:22:30Z","updated_at":"2015-12-14T20:22:30Z","html_url":"https://github.com/apache/spark/pull/9765#discussion_r47552258","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/9765","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/47552258"},"html":{"href":"https://github.com/apache/spark/pull/9765#discussion_r47552258"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/9765"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9765","id":50922514,"html_url":"https://github.com/apache/spark/pull/9765","diff_url":"https://github.com/apache/spark/pull/9765.diff","patch_url":"https://github.com/apache/spark/pull/9765.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/9765","number":9765,"state":"open","locked":false,"title":"[SPARK-11749][Streaming] Duplicate creating the RDD in file stream when recovering from checkpoint data","user":{"login":"jhu-chang","id":10203485,"avatar_url":"https://avatars.githubusercontent.com/u/10203485?v=3","gravatar_id":"","url":"https://api.github.com/users/jhu-chang","html_url":"https://github.com/jhu-chang","followers_url":"https://api.github.com/users/jhu-chang/followers","following_url":"https://api.github.com/users/jhu-chang/following{/other_user}","gists_url":"https://api.github.com/users/jhu-chang/gists{/gist_id}","starred_url":"https://api.github.com/users/jhu-chang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhu-chang/subscriptions","organizations_url":"https://api.github.com/users/jhu-chang/orgs","repos_url":"https://api.github.com/users/jhu-chang/repos","events_url":"https://api.github.com/users/jhu-chang/events{/privacy}","received_events_url":"https://api.github.com/users/jhu-chang/received_events","type":"User","site_admin":false},"body":"Add a transient flag `DStream.restoredFromCheckpointData` to control the restore processing in DStream to avoid duplicate works:  check this flag first in `DStream.restoreCheckpointData`, only when `false`, the restore process will be executed.  \r\n","created_at":"2015-11-17T09:31:59Z","updated_at":"2015-12-14T20:22:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"51fd5a2fcb31b1a360923f754cbe199f0cae7d9f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/9765/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/9765/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/9765/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/ac8795197f6fa552b4215c63c93b6112652972d9","head":{"label":"jhu-chang:SPARK-11749","ref":"SPARK-11749","sha":"ac8795197f6fa552b4215c63c93b6112652972d9","user":{"login":"jhu-chang","id":10203485,"avatar_url":"https://avatars.githubusercontent.com/u/10203485?v=3","gravatar_id":"","url":"https://api.github.com/users/jhu-chang","html_url":"https://github.com/jhu-chang","followers_url":"https://api.github.com/users/jhu-chang/followers","following_url":"https://api.github.com/users/jhu-chang/following{/other_user}","gists_url":"https://api.github.com/users/jhu-chang/gists{/gist_id}","starred_url":"https://api.github.com/users/jhu-chang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhu-chang/subscriptions","organizations_url":"https://api.github.com/users/jhu-chang/orgs","repos_url":"https://api.github.com/users/jhu-chang/repos","events_url":"https://api.github.com/users/jhu-chang/events{/privacy}","received_events_url":"https://api.github.com/users/jhu-chang/received_events","type":"User","site_admin":false},"repo":{"id":32568939,"name":"spark","full_name":"jhu-chang/spark","owner":{"login":"jhu-chang","id":10203485,"avatar_url":"https://avatars.githubusercontent.com/u/10203485?v=3","gravatar_id":"","url":"https://api.github.com/users/jhu-chang","html_url":"https://github.com/jhu-chang","followers_url":"https://api.github.com/users/jhu-chang/followers","following_url":"https://api.github.com/users/jhu-chang/following{/other_user}","gists_url":"https://api.github.com/users/jhu-chang/gists{/gist_id}","starred_url":"https://api.github.com/users/jhu-chang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhu-chang/subscriptions","organizations_url":"https://api.github.com/users/jhu-chang/orgs","repos_url":"https://api.github.com/users/jhu-chang/repos","events_url":"https://api.github.com/users/jhu-chang/events{/privacy}","received_events_url":"https://api.github.com/users/jhu-chang/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jhu-chang/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/jhu-chang/spark","forks_url":"https://api.github.com/repos/jhu-chang/spark/forks","keys_url":"https://api.github.com/repos/jhu-chang/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jhu-chang/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jhu-chang/spark/teams","hooks_url":"https://api.github.com/repos/jhu-chang/spark/hooks","issue_events_url":"https://api.github.com/repos/jhu-chang/spark/issues/events{/number}","events_url":"https://api.github.com/repos/jhu-chang/spark/events","assignees_url":"https://api.github.com/repos/jhu-chang/spark/assignees{/user}","branches_url":"https://api.github.com/repos/jhu-chang/spark/branches{/branch}","tags_url":"https://api.github.com/repos/jhu-chang/spark/tags","blobs_url":"https://api.github.com/repos/jhu-chang/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jhu-chang/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jhu-chang/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/jhu-chang/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jhu-chang/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/jhu-chang/spark/languages","stargazers_url":"https://api.github.com/repos/jhu-chang/spark/stargazers","contributors_url":"https://api.github.com/repos/jhu-chang/spark/contributors","subscribers_url":"https://api.github.com/repos/jhu-chang/spark/subscribers","subscription_url":"https://api.github.com/repos/jhu-chang/spark/subscription","commits_url":"https://api.github.com/repos/jhu-chang/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/jhu-chang/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/jhu-chang/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/jhu-chang/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/jhu-chang/spark/contents/{+path}","compare_url":"https://api.github.com/repos/jhu-chang/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jhu-chang/spark/merges","archive_url":"https://api.github.com/repos/jhu-chang/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jhu-chang/spark/downloads","issues_url":"https://api.github.com/repos/jhu-chang/spark/issues{/number}","pulls_url":"https://api.github.com/repos/jhu-chang/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/jhu-chang/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/jhu-chang/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jhu-chang/spark/labels{/name}","releases_url":"https://api.github.com/repos/jhu-chang/spark/releases{/id}","created_at":"2015-03-20T07:19:19Z","updated_at":"2015-03-23T03:04:30Z","pushed_at":"2015-12-14T09:07:18Z","git_url":"git://github.com/jhu-chang/spark.git","ssh_url":"git@github.com:jhu-chang/spark.git","clone_url":"https://github.com/jhu-chang/spark.git","svn_url":"https://github.com/jhu-chang/spark","homepage":null,"size":133564,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"8af2f8c61ae4a59d129fb3530d0f6e9317f4bff8","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-12-14T18:21:11Z","pushed_at":"2015-12-14T19:23:13Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":391380,"stargazers_count":6574,"watchers_count":6574,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":6090,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":415,"forks":6090,"open_issues":415,"watchers":6574,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/9765"},"html":{"href":"https://github.com/apache/spark/pull/9765"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/9765"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/9765/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/9765/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/9765/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/ac8795197f6fa552b4215c63c93b6112652972d9"}}}}