{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8877","labels_url":"https://api.github.com/repos/apache/spark/issues/8877/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8877/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8877/events","html_url":"https://github.com/apache/spark/pull/8877","id":107853707,"number":8877,"title":"[SPARK-10769][Streaming][Tests]Fix o.a.s.streaming.CheckpointSuite.maintains rate controller","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-23T05:43:33Z","updated_at":"2015-09-23T06:15:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8877","html_url":"https://github.com/apache/spark/pull/8877","diff_url":"https://github.com/apache/spark/pull/8877.diff","patch_url":"https://github.com/apache/spark/pull/8877.patch"},"body":"Fixed the following failure in https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1787/testReport/junit/org.apache.spark.streaming/CheckpointSuite/recovery_maintains_rate_controller/\r\n```\r\nsbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 660 times over 10.000044392000001 seconds. Last failure message: 9223372036854775807 did not equal 200.\r\n\tat org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\r\n\tat org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\r\n\tat org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\r\n\tat org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:336)\r\n\tat org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\r\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply$mcV$sp(CheckpointSuite.scala:413)\r\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply(CheckpointSuite.scala:396)\r\n\tat org.apache.spark.streaming.CheckpointSuite$$anonfun$15.apply(CheckpointSuite.scala:396)\r\n\tat org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\r\n\tat org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\r\n\tat org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\r\n\tat org.scalatest.Transformer.apply(Transformer.scala:22)\r\n```\r\n\r\nIn this test, it calls `advanceTimeWithRealDelay(ssc, 2)` to run two batch jobs. However, one race condition is these two jobs can finish before the receiver is registered. Then `UpdateRateLimit` won't be sent to the receiver and `getDefaultBlockGeneratorRateLimit` cannot be updated.\r\n\r\nHere are the logs related to this issue:\r\n```\r\n15/09/22 19:28:26.154 pool-1-thread-1-ScalaTest-running-CheckpointSuite INFO CheckpointSuite: Manual clock before advancing = 2500\r\n\r\n15/09/22 19:28:26.869 JobScheduler INFO JobScheduler: Finished job streaming job 3000 ms.0 from job set of time 3000 ms\r\n15/09/22 19:28:26.869 JobScheduler INFO JobScheduler: Total delay: 1442975303.869 s for time 3000 ms (execution: 0.711 s)\r\n\r\n15/09/22 19:28:26.873 JobScheduler INFO JobScheduler: Finished job streaming job 3500 ms.0 from job set of time 3500 ms\r\n15/09/22 19:28:26.873 JobScheduler INFO JobScheduler: Total delay: 1442975303.373 s for time 3500 ms (execution: 0.004 s)\r\n\r\n15/09/22 19:28:26.879 sparkDriver-akka.actor.default-dispatcher-3 INFO ReceiverTracker: Registered receiver for stream 0 from localhost:57749\r\n\r\n15/09/22 19:28:27.154 pool-1-thread-1-ScalaTest-running-CheckpointSuite INFO CheckpointSuite: Manual clock after advancing = 3500\r\n```\r\n`advanceTimeWithRealDelay(ssc, 2)` triggered job 3000ms and 3500ms but the receiver was registered after job 3000ms and 3500ms finished.\r\n\r\nSo we should make sure the receiver online before running `advanceTimeWithRealDelay(ssc, 2)`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/142505271","html_url":"https://github.com/apache/spark/pull/8877#issuecomment-142505271","issue_url":"https://api.github.com/repos/apache/spark/issues/8877","id":142505271,"user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T06:15:12Z","updated_at":"2015-09-23T06:15:12Z","body":"test this again"}}