{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4337","labels_url":"https://api.github.com/repos/apache/spark/issues/4337/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4337/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4337/events","html_url":"https://github.com/apache/spark/pull/4337","id":56382190,"number":4337,"title":"[SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite","user":{"login":"sarutak","id":4736016,"avatar_url":"https://avatars.githubusercontent.com/u/4736016?v=3","gravatar_id":"","url":"https://api.github.com/users/sarutak","html_url":"https://github.com/sarutak","followers_url":"https://api.github.com/users/sarutak/followers","following_url":"https://api.github.com/users/sarutak/following{/other_user}","gists_url":"https://api.github.com/users/sarutak/gists{/gist_id}","starred_url":"https://api.github.com/users/sarutak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sarutak/subscriptions","organizations_url":"https://api.github.com/users/sarutak/orgs","repos_url":"https://api.github.com/users/sarutak/repos","events_url":"https://api.github.com/users/sarutak/events{/privacy}","received_events_url":"https://api.github.com/users/sarutak/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2015-02-03T14:11:37Z","updated_at":"2015-03-23T19:40:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4337","html_url":"https://github.com/apache/spark/pull/4337","diff_url":"https://github.com/apache/spark/pull/4337.diff","patch_url":"https://github.com/apache/spark/pull/4337.patch"},"body":"When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows.\r\n\r\n\r\n    sbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 52 times over 10.094849836 seconds. Last failure message: Error connecting to localhost/127.0.0.1:23456.\r\n\t    at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)\r\n\t    at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)\r\n\t    at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\r\n\t    at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)\r\n\t   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)\r\n\t   at org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116)\r\n           at org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74)\r\n\t   at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66)\r\n\t    at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)\r\n\t    at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)\r\n\t    at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)\r\n\t    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)\r\n\t    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\r\n\t    at org.scalatest.Transformer.apply(Transformer.scala:22)\r\n\t    at org.scalatest.Transformer.apply(Transformer.scala:20)\r\n    \t    at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)\r\n\t    at org.scalatest.Suite$class.withFixture(Suite.scala:1122)\r\n\t    at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)\r\n\t    at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)\r\n\t   at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\r\n\t    at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)\r\n\t    at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)\r\n\t    at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)\r\n\r\n\r\nThis error is caused by check-then-act logic  when it find free-port .\r\n\r\n\r\n      /** Find a free port */\r\n      private def findFreePort(): Int = {\r\n        Utils.startServiceOnPort(23456, (trialPort: Int) => {\r\n          val socket = new ServerSocket(trialPort)\r\n          socket.close()\r\n          (null, trialPort)\r\n        }, conf)._2\r\n      }\r\n\r\n\r\nRemoving the check-then-act is not easy but we can reduce the chance of having the error by choosing random value for initial port instead of 23456."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/85163303","html_url":"https://github.com/apache/spark/pull/4337#issuecomment-85163303","issue_url":"https://api.github.com/repos/apache/spark/issues/4337","id":85163303,"user":{"login":"srowen","id":822522,"avatar_url":"https://avatars.githubusercontent.com/u/822522?v=3","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"created_at":"2015-03-23T19:40:46Z","updated_at":"2015-03-23T19:40:46Z","body":"OK I think the import is still in slightly the wrong place in both cases and the text could be edited a bit, but I can carefully adjust that on merge."}}