{"action":"created","issue":{"url":"https://api.github.com/repos/kframework/k/issues/1959","labels_url":"https://api.github.com/repos/kframework/k/issues/1959/labels{/name}","comments_url":"https://api.github.com/repos/kframework/k/issues/1959/comments","events_url":"https://api.github.com/repos/kframework/k/issues/1959/events","html_url":"https://github.com/kframework/k/issues/1959","id":124144075,"number":1959,"title":"Flaky tests","user":{"login":"cos","id":37514,"avatar_url":"https://avatars.githubusercontent.com/u/37514?v=3","gravatar_id":"","url":"https://api.github.com/users/cos","html_url":"https://github.com/cos","followers_url":"https://api.github.com/users/cos/followers","following_url":"https://api.github.com/users/cos/following{/other_user}","gists_url":"https://api.github.com/users/cos/gists{/gist_id}","starred_url":"https://api.github.com/users/cos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cos/subscriptions","organizations_url":"https://api.github.com/users/cos/orgs","repos_url":"https://api.github.com/users/cos/repos","events_url":"https://api.github.com/users/cos/events{/privacy}","received_events_url":"https://api.github.com/users/cos/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kframework/k/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"cos","id":37514,"avatar_url":"https://avatars.githubusercontent.com/u/37514?v=3","gravatar_id":"","url":"https://api.github.com/users/cos","html_url":"https://github.com/cos","followers_url":"https://api.github.com/users/cos/followers","following_url":"https://api.github.com/users/cos/following{/other_user}","gists_url":"https://api.github.com/users/cos/gists{/gist_id}","starred_url":"https://api.github.com/users/cos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cos/subscriptions","organizations_url":"https://api.github.com/users/cos/orgs","repos_url":"https://api.github.com/users/cos/repos","events_url":"https://api.github.com/users/cos/events{/privacy}","received_events_url":"https://api.github.com/users/cos/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kframework/k/milestones/12","html_url":"https://github.com/kframework/k/milestones/4.0%20beta","labels_url":"https://api.github.com/repos/kframework/k/milestones/12/labels","id":1435572,"number":12,"title":"4.0 beta","description":"","creator":{"login":"cos","id":37514,"avatar_url":"https://avatars.githubusercontent.com/u/37514?v=3","gravatar_id":"","url":"https://api.github.com/users/cos","html_url":"https://github.com/cos","followers_url":"https://api.github.com/users/cos/followers","following_url":"https://api.github.com/users/cos/following{/other_user}","gists_url":"https://api.github.com/users/cos/gists{/gist_id}","starred_url":"https://api.github.com/users/cos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cos/subscriptions","organizations_url":"https://api.github.com/users/cos/orgs","repos_url":"https://api.github.com/users/cos/repos","events_url":"https://api.github.com/users/cos/events{/privacy}","received_events_url":"https://api.github.com/users/cos/received_events","type":"User","site_admin":false},"open_issues":24,"closed_issues":10,"state":"open","created_at":"2015-11-30T16:25:24Z","updated_at":"2015-12-29T22:23:20Z","due_on":"2016-01-06T06:00:00Z","closed_at":null},"comments":3,"created_at":"2015-12-29T00:15:59Z","updated_at":"2015-12-29T22:41:30Z","closed_at":null,"body":"The `ktest`s are flaky, i.e., they fail non-deterministically -- true failures, not printing issues. For now, the problem is not that bad and in most cases a retry via \"Jenkins: test this please\" solves the problem. Still, as we add more of the tutorial, the problem will likely get worse.\r\n\r\nLet's use this issue to mark the tests that we find flaky so that someone can inspect them at some point. I forgot to note down the tests as I encountered them but will do so from now on. Hopefully the problems are in the definitions and can be easily identified via `search`.\r\n\r\nIf you feel like looking into this problem, feel free to pick up this issue.\r\n\r\nnotifying active developers: @andreistefanescu, @daejunpark, @laurayuwen, @bmmoore, @dwightguth\r\n\r\nList of non-deterministic tests (edit this comment to add more): \r\n- http://office.runtimeverification.com:8080/job/k-framework-pull-request/423/ \r\n`\r\n/var/lib/jenkins/jobs/k-framework-pull-request/workspace/k-distribution/tutorial/1_k/3_lambda++/lesson_5.callcc-not-jump\r\n`"},"comment":{"url":"https://api.github.com/repos/kframework/k/issues/comments/167893799","html_url":"https://github.com/kframework/k/issues/1959#issuecomment-167893799","issue_url":"https://api.github.com/repos/kframework/k/issues/1959","id":167893799,"user":{"login":"cos","id":37514,"avatar_url":"https://avatars.githubusercontent.com/u/37514?v=3","gravatar_id":"","url":"https://api.github.com/users/cos","html_url":"https://github.com/cos","followers_url":"https://api.github.com/users/cos/followers","following_url":"https://api.github.com/users/cos/following{/other_user}","gists_url":"https://api.github.com/users/cos/gists{/gist_id}","starred_url":"https://api.github.com/users/cos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cos/subscriptions","organizations_url":"https://api.github.com/users/cos/orgs","repos_url":"https://api.github.com/users/cos/repos","events_url":"https://api.github.com/users/cos/events{/privacy}","received_events_url":"https://api.github.com/users/cos/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T22:41:30Z","updated_at":"2015-12-29T22:41:30Z","body":"Also, regarding 1): those are likely not low-level races but higher-level atomicity violations, either at the level of Java code, or even non-confluence in the definitions themselves.\r\n"}}