{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3687","labels_url":"https://api.github.com/repos/apache/spark/issues/3687/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3687/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3687/events","html_url":"https://github.com/apache/spark/pull/3687","id":51872025,"number":3687,"title":"[WIP] Remove many uses of Thread.sleep() from streaming tests","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2014-12-13T01:41:29Z","updated_at":"2015-01-07T19:47:50Z","closed_at":"2015-01-07T19:47:50Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3687","html_url":"https://github.com/apache/spark/pull/3687","diff_url":"https://github.com/apache/spark/pull/3687.diff","patch_url":"https://github.com/apache/spark/pull/3687.patch"},"body":"**Update:** we have decided to split this into a series of smaller PRs: #3801 and #3832.\r\n\r\nThis is a work-in-progress PR towards removing many of the `Thread.sleep()` calls from Spark Streaming's test suite, since I think that these calls make these tests race-prone and flaky.\r\n\r\n**Running list of the JIRAs that this patch addresses / intends to address**:\r\n\r\n- [SPARK-4835](https://issues.apache.org/jira/browse/SPARK-4835): \"Streaming saveAs*HadoopFiles() methods may throw FileAlreadyExistsException during checkpoint recovery\"\r\n- [SPARK-1600](https://issues.apache.org/jira/browse/SPARK-1600): \"Flaky \"recovery with file input stream\" test in streaming.CheckpointSuite\"\r\n\r\n**TODOS**:\r\n\r\n- [x] Fix SPARK-1600 by removing the `Thread.sleep` calls from the `recovery with file input stream` test.  There are several uses of `Thread.sleep` here which seem to serve several different purposes, so this will require some care.\r\n- [x] Remove the sleep calls in `InputStreamSuite`.  Most of these cases look simple to replace with the new StreamingTestWaiter, but the `testFileStream` case looks like it could be tricky.  Going to defer removing the sleeps from `multi-thread receiver` since that looks like a bit of work and we can do it later if we observe flakiness.\r\n- [ ] Investigate the uses in `ReceiverSuite`.  There are a few `Thread.sleep(0)` cases which seem confusing; if they're still necessary, they should be commented.\r\n- [ ] Might be able to remove one or two low-hanging fruit calls in `MasterFailureTest`, since these seem to be blocking on events like the StreamingContext starting and we have waiters for this.\r\n- [ ] Determine whether the fix for SPARK-4835 should be done differently (see discussion on JIRA)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/69080052","html_url":"https://github.com/apache/spark/pull/3687#issuecomment-69080052","issue_url":"https://api.github.com/repos/apache/spark/issues/3687","id":69080052,"user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T19:47:49Z","updated_at":"2015-01-07T19:47:49Z","body":"I'm going to close this for now; I'll open separate, smaller PRs for the remaining test cleanup."}}