{"action":"created","issue":{"url":"https://api.github.com/repos/klarna/HiveRunner/issues/11","labels_url":"https://api.github.com/repos/klarna/HiveRunner/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/klarna/HiveRunner/issues/11/comments","events_url":"https://api.github.com/repos/klarna/HiveRunner/issues/11/events","html_url":"https://github.com/klarna/HiveRunner/issues/11","id":111841033,"number":11,"title":"TimeoutAndRetryTest test suite is nondeterministic","user":{"login":"teabot","id":228950,"avatar_url":"https://avatars.githubusercontent.com/u/228950?v=3","gravatar_id":"","url":"https://api.github.com/users/teabot","html_url":"https://github.com/teabot","followers_url":"https://api.github.com/users/teabot/followers","following_url":"https://api.github.com/users/teabot/following{/other_user}","gists_url":"https://api.github.com/users/teabot/gists{/gist_id}","starred_url":"https://api.github.com/users/teabot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teabot/subscriptions","organizations_url":"https://api.github.com/users/teabot/orgs","repos_url":"https://api.github.com/users/teabot/repos","events_url":"https://api.github.com/users/teabot/events{/privacy}","received_events_url":"https://api.github.com/users/teabot/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-16T14:14:58Z","updated_at":"2015-10-16T15:08:25Z","closed_at":null,"body":"The `com.klarna.hiverrunnrer.TimeoutAndRetryTest` seems to fail/pass depending on the time it takes to execute. Increasing the timeout on the tests make them more likely to pass. In other environments the tests seem to pass consistently without issue.\r\n\r\nOutput from sample runs on the same codebase:\r\n\r\n#### 1.\r\n````\r\nTests in error:\r\nTimeoutAndRetryTest.endOnSecondRun » Timeout test timed out after 15 seconds\r\nTimeoutAndRetryTest.expectTest » Timeout test timed out after 15 seconds\r\nTimeoutAndRetryTest.throwOnSecondRun2 » Timeout test timed out after 15 second...\r\nTimeoutAndRetryTest.expectTimoutTest » Timeout test timed out after 15 seconds\r\nTimeoutAndRetryTest.throwOnSecondRun » Timeout test timed out after 15 seconds\r\n````\r\n\r\n#### 2.\r\n````\r\nResults :\r\nTests run: 6, Failures: 0, Errors: 0, Skipped: 1\r\n````\r\n\r\n#### 3.\r\n````\r\nTests in error:\r\n  TimeoutAndRetryTest.throwOnSecondRun » Timeout test timed out after 15 seconds\r\nTests run: 6, Failures: 0, Errors: 1, Skipped: 1\r\n````"},"comment":{"url":"https://api.github.com/repos/klarna/HiveRunner/issues/comments/148741871","html_url":"https://github.com/klarna/HiveRunner/issues/11#issuecomment-148741871","issue_url":"https://api.github.com/repos/klarna/HiveRunner/issues/11","id":148741871,"user":{"login":"PelleUllberg","id":5992231,"avatar_url":"https://avatars.githubusercontent.com/u/5992231?v=3","gravatar_id":"","url":"https://api.github.com/users/PelleUllberg","html_url":"https://github.com/PelleUllberg","followers_url":"https://api.github.com/users/PelleUllberg/followers","following_url":"https://api.github.com/users/PelleUllberg/following{/other_user}","gists_url":"https://api.github.com/users/PelleUllberg/gists{/gist_id}","starred_url":"https://api.github.com/users/PelleUllberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PelleUllberg/subscriptions","organizations_url":"https://api.github.com/users/PelleUllberg/orgs","repos_url":"https://api.github.com/users/PelleUllberg/repos","events_url":"https://api.github.com/users/PelleUllberg/events{/privacy}","received_events_url":"https://api.github.com/users/PelleUllberg/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T15:08:25Z","updated_at":"2015-10-16T15:08:25Z","body":"Hi, yes that is the nature of those tests unless you uncovered a bug... \r\n\r\nTime based tests are sketchy and should be avoided if possible. However since there is now a timeout feature in HiveRunner I also wanted to add some unit tests for that feature. 15s was enough on my machine and on our continuous build environment but maybe we should bump that a bit to accommodate for slower environments. The other option is to put them in a separate test suite or just tag them with @Ignore. \r\n\r\nThe boring part with bumping the timeout is that the full test suite will take longer to run. \r\n\r\nCan you find a stable setting for your environments as input to what it should be set to?\r\n\r\nregards\r\n/Pelle"}}