{"action":"created","comment":{"url":"https://api.github.com/repos/mono/mono/pulls/comments/48056916","id":48056916,"diff_hunk":"@@ -0,0 +1,104 @@\n+// Added to NUnit by Andi McClure to add special support for the test babysitter script.\n+// See scripts/babysitter in Mono repository.\n+\n+namespace NUnit.Core\n+{\n+\tusing System;\n+\tusing System.IO;\n+\tusing System.Collections.Generic;\n+\tusing NUnit.Core.Filters;\n+\n+\tpublic class BabysitterSupport\n+\t{\n+\t\tenum OverrideMode {None, Run, Exclude};\n+\t\tpublic static Dictionary<string, bool> OverrideTests = new Dictionary<string, bool>();\n+\t\tprivate static OverrideMode Override = OverrideMode.None;\n+\t\tprivate static string CurrentTestFile = null, RanTestFile = null, FailedTestFile = null;\n+\n+\t\tprivate static void DeleteFile(string path)\n+\t\t{\n+\t\t\ttry {\n+\t\t\t\tFile.Delete(path);\n+\t\t\t} catch (Exception e) {}\n+\t\t}\n+\t\tprivate static void WriteFile(string path, string contents)\n+\t\t{\n+\t\t\tDeleteFile(path);\n+\t\t\tFile.AppendAllText(path, contents);\n+\t\t}\n+\n+\t\t// Environment variables are available from process start, so safe to do setup in a static constructor\n+\t\tstatic BabysitterSupport()\n+\t\t{\n+\t\t\tstring overrideModeString = Environment.GetEnvironmentVariable(\"NUNIT_BABYSITTER_RUN_MODE\");\n+\t\t\tstring overrideTestString = Environment.GetEnvironmentVariable(\"NUNIT_BABYSITTER_RUN_TEST\");\n+\t\t\tif (overrideModeString == \"RUN\")\n+\t\t\t\tOverride = OverrideMode.Run;\n+\t\t\telse if (overrideModeString == \"EXCLUDE\")\n+\t\t\t\tOverride = OverrideMode.Exclude;\n+\t\t\tif (Override != OverrideMode.None)\n+\t\t\t{\n+\t\t\t\tchar [] semicolon = {';'};\n+\t\t\t\tforeach (string s in overrideTestString.Split(semicolon, StringSplitOptions.RemoveEmptyEntries))","path":"mcs/nunit24/NUnitCore/core/BabysitterSupport.cs","position":42,"original_position":42,"commit_id":"1dc0598badaed161efa92fd261de7e555d1a2b13","original_commit_id":"1dc0598badaed161efa92fd261de7e555d1a2b13","user":{"login":"akoeplinger","id":1376924,"avatar_url":"https://avatars.githubusercontent.com/u/1376924?v=3","gravatar_id":"","url":"https://api.github.com/users/akoeplinger","html_url":"https://github.com/akoeplinger","followers_url":"https://api.github.com/users/akoeplinger/followers","following_url":"https://api.github.com/users/akoeplinger/following{/other_user}","gists_url":"https://api.github.com/users/akoeplinger/gists{/gist_id}","starred_url":"https://api.github.com/users/akoeplinger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akoeplinger/subscriptions","organizations_url":"https://api.github.com/users/akoeplinger/orgs","repos_url":"https://api.github.com/users/akoeplinger/repos","events_url":"https://api.github.com/users/akoeplinger/events{/privacy}","received_events_url":"https://api.github.com/users/akoeplinger/received_events","type":"User","site_admin":false},"body":"don't think semicolon warrants a separate variable.","created_at":"2015-12-18T18:57:10Z","updated_at":"2015-12-18T18:57:10Z","html_url":"https://github.com/mono/mono/pull/2366#discussion_r48056916","pull_request_url":"https://api.github.com/repos/mono/mono/pulls/2366","_links":{"self":{"href":"https://api.github.com/repos/mono/mono/pulls/comments/48056916"},"html":{"href":"https://github.com/mono/mono/pull/2366#discussion_r48056916"},"pull_request":{"href":"https://api.github.com/repos/mono/mono/pulls/2366"}}},"pull_request":{"url":"https://api.github.com/repos/mono/mono/pulls/2366","id":54134527,"html_url":"https://github.com/mono/mono/pull/2366","diff_url":"https://github.com/mono/mono/pull/2366.diff","patch_url":"https://github.com/mono/mono/pull/2366.patch","issue_url":"https://api.github.com/repos/mono/mono/issues/2366","number":2366,"state":"open","locked":false,"title":"Adds babysitter script for PR tests","user":{"login":"xmcclure","id":15091917,"avatar_url":"https://avatars.githubusercontent.com/u/15091917?v=3","gravatar_id":"","url":"https://api.github.com/users/xmcclure","html_url":"https://github.com/xmcclure","followers_url":"https://api.github.com/users/xmcclure/followers","following_url":"https://api.github.com/users/xmcclure/following{/other_user}","gists_url":"https://api.github.com/users/xmcclure/gists{/gist_id}","starred_url":"https://api.github.com/users/xmcclure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xmcclure/subscriptions","organizations_url":"https://api.github.com/users/xmcclure/orgs","repos_url":"https://api.github.com/users/xmcclure/repos","events_url":"https://api.github.com/users/xmcclure/events{/privacy}","received_events_url":"https://api.github.com/users/xmcclure/received_events","type":"User","site_admin":false},"body":"I see two problems with our current automated PR tests:\r\n- Many false negatives— PR tests often fail due to recurring failures unrelated to the PR\r\n- Information about what failed is awkward to retrieve, scattered in multiple places\r\n\r\nThis commit adds a Python script, `scripts/babysitter`. Currently our Jenkins test suites are each run wrapped with the `timeout` command. The babysitter acts as a drop-in replacement for GNU timeout but adds the following features:\r\n- Logs machine readable output about each test suite (as line-delimited json)\r\n- If the test suite uses NUnit, can detect if a test case failed or crashed (terminated mono in mid-test), and retry unsuccessful tests (up to a limit).\r\n\r\nThe reasoning here is that tests which fail inconsistently are currently most likely due to one of our outstanding recurring failures rather than the change made in the PR. Therefore, if a failing test\r\nsucceeds on retry, the PR itself is probably valid (although the failure should be logged and looked at).\r\n\r\nIn addition to the script itself, changes to NUnit were required in order to support the retry feature and allow fine-grained logging for NUnit suites.\r\n\r\nMajor TODOs:\r\n- Add retry support for our non-NUnit-based test suites\r\n- Save the XML files NUnit produces (since reruns stomp XMLs from previous runs)\r\n- Add some kind of sensible feature for dealing with timeouts","created_at":"2015-12-18T18:31:37Z","updated_at":"2015-12-18T18:57:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a136b86f32982cb0991c06b45498a2a233c0efa8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mono/mono/pulls/2366/commits","review_comments_url":"https://api.github.com/repos/mono/mono/pulls/2366/comments","review_comment_url":"https://api.github.com/repos/mono/mono/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mono/mono/issues/2366/comments","statuses_url":"https://api.github.com/repos/mono/mono/statuses/1dc0598badaed161efa92fd261de7e555d1a2b13","head":{"label":"xmcclure:scripts-babysitter","ref":"scripts-babysitter","sha":"1dc0598badaed161efa92fd261de7e555d1a2b13","user":{"login":"xmcclure","id":15091917,"avatar_url":"https://avatars.githubusercontent.com/u/15091917?v=3","gravatar_id":"","url":"https://api.github.com/users/xmcclure","html_url":"https://github.com/xmcclure","followers_url":"https://api.github.com/users/xmcclure/followers","following_url":"https://api.github.com/users/xmcclure/following{/other_user}","gists_url":"https://api.github.com/users/xmcclure/gists{/gist_id}","starred_url":"https://api.github.com/users/xmcclure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xmcclure/subscriptions","organizations_url":"https://api.github.com/users/xmcclure/orgs","repos_url":"https://api.github.com/users/xmcclure/repos","events_url":"https://api.github.com/users/xmcclure/events{/privacy}","received_events_url":"https://api.github.com/users/xmcclure/received_events","type":"User","site_admin":false},"repo":{"id":44119985,"name":"mono","full_name":"xmcclure/mono","owner":{"login":"xmcclure","id":15091917,"avatar_url":"https://avatars.githubusercontent.com/u/15091917?v=3","gravatar_id":"","url":"https://api.github.com/users/xmcclure","html_url":"https://github.com/xmcclure","followers_url":"https://api.github.com/users/xmcclure/followers","following_url":"https://api.github.com/users/xmcclure/following{/other_user}","gists_url":"https://api.github.com/users/xmcclure/gists{/gist_id}","starred_url":"https://api.github.com/users/xmcclure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xmcclure/subscriptions","organizations_url":"https://api.github.com/users/xmcclure/orgs","repos_url":"https://api.github.com/users/xmcclure/repos","events_url":"https://api.github.com/users/xmcclure/events{/privacy}","received_events_url":"https://api.github.com/users/xmcclure/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/xmcclure/mono","description":"Mono open source ECMA CLI, C# and .NET implementation.","fork":true,"url":"https://api.github.com/repos/xmcclure/mono","forks_url":"https://api.github.com/repos/xmcclure/mono/forks","keys_url":"https://api.github.com/repos/xmcclure/mono/keys{/key_id}","collaborators_url":"https://api.github.com/repos/xmcclure/mono/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/xmcclure/mono/teams","hooks_url":"https://api.github.com/repos/xmcclure/mono/hooks","issue_events_url":"https://api.github.com/repos/xmcclure/mono/issues/events{/number}","events_url":"https://api.github.com/repos/xmcclure/mono/events","assignees_url":"https://api.github.com/repos/xmcclure/mono/assignees{/user}","branches_url":"https://api.github.com/repos/xmcclure/mono/branches{/branch}","tags_url":"https://api.github.com/repos/xmcclure/mono/tags","blobs_url":"https://api.github.com/repos/xmcclure/mono/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/xmcclure/mono/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/xmcclure/mono/git/refs{/sha}","trees_url":"https://api.github.com/repos/xmcclure/mono/git/trees{/sha}","statuses_url":"https://api.github.com/repos/xmcclure/mono/statuses/{sha}","languages_url":"https://api.github.com/repos/xmcclure/mono/languages","stargazers_url":"https://api.github.com/repos/xmcclure/mono/stargazers","contributors_url":"https://api.github.com/repos/xmcclure/mono/contributors","subscribers_url":"https://api.github.com/repos/xmcclure/mono/subscribers","subscription_url":"https://api.github.com/repos/xmcclure/mono/subscription","commits_url":"https://api.github.com/repos/xmcclure/mono/commits{/sha}","git_commits_url":"https://api.github.com/repos/xmcclure/mono/git/commits{/sha}","comments_url":"https://api.github.com/repos/xmcclure/mono/comments{/number}","issue_comment_url":"https://api.github.com/repos/xmcclure/mono/issues/comments{/number}","contents_url":"https://api.github.com/repos/xmcclure/mono/contents/{+path}","compare_url":"https://api.github.com/repos/xmcclure/mono/compare/{base}...{head}","merges_url":"https://api.github.com/repos/xmcclure/mono/merges","archive_url":"https://api.github.com/repos/xmcclure/mono/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/xmcclure/mono/downloads","issues_url":"https://api.github.com/repos/xmcclure/mono/issues{/number}","pulls_url":"https://api.github.com/repos/xmcclure/mono/pulls{/number}","milestones_url":"https://api.github.com/repos/xmcclure/mono/milestones{/number}","notifications_url":"https://api.github.com/repos/xmcclure/mono/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/xmcclure/mono/labels{/name}","releases_url":"https://api.github.com/repos/xmcclure/mono/releases{/id}","created_at":"2015-10-12T16:32:27Z","updated_at":"2015-10-12T16:35:38Z","pushed_at":"2015-12-18T18:30:48Z","git_url":"git://github.com/xmcclure/mono.git","ssh_url":"git@github.com:xmcclure/mono.git","clone_url":"https://github.com/xmcclure/mono.git","svn_url":"https://github.com/xmcclure/mono","homepage":"http://www.mono-project.com/","size":453604,"stargazers_count":0,"watchers_count":0,"language":"C#","has_issues":false,"has_downloads":false,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mono:master","ref":"master","sha":"e20cd1cf85333d8ac1e96a7d20585f2561101ffe","user":{"login":"mono","id":53395,"avatar_url":"https://avatars.githubusercontent.com/u/53395?v=3","gravatar_id":"","url":"https://api.github.com/users/mono","html_url":"https://github.com/mono","followers_url":"https://api.github.com/users/mono/followers","following_url":"https://api.github.com/users/mono/following{/other_user}","gists_url":"https://api.github.com/users/mono/gists{/gist_id}","starred_url":"https://api.github.com/users/mono/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mono/subscriptions","organizations_url":"https://api.github.com/users/mono/orgs","repos_url":"https://api.github.com/users/mono/repos","events_url":"https://api.github.com/users/mono/events{/privacy}","received_events_url":"https://api.github.com/users/mono/received_events","type":"Organization","site_admin":false},"repo":{"id":791611,"name":"mono","full_name":"mono/mono","owner":{"login":"mono","id":53395,"avatar_url":"https://avatars.githubusercontent.com/u/53395?v=3","gravatar_id":"","url":"https://api.github.com/users/mono","html_url":"https://github.com/mono","followers_url":"https://api.github.com/users/mono/followers","following_url":"https://api.github.com/users/mono/following{/other_user}","gists_url":"https://api.github.com/users/mono/gists{/gist_id}","starred_url":"https://api.github.com/users/mono/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mono/subscriptions","organizations_url":"https://api.github.com/users/mono/orgs","repos_url":"https://api.github.com/users/mono/repos","events_url":"https://api.github.com/users/mono/events{/privacy}","received_events_url":"https://api.github.com/users/mono/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mono/mono","description":"Mono open source ECMA CLI, C# and .NET implementation.","fork":false,"url":"https://api.github.com/repos/mono/mono","forks_url":"https://api.github.com/repos/mono/mono/forks","keys_url":"https://api.github.com/repos/mono/mono/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mono/mono/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mono/mono/teams","hooks_url":"https://api.github.com/repos/mono/mono/hooks","issue_events_url":"https://api.github.com/repos/mono/mono/issues/events{/number}","events_url":"https://api.github.com/repos/mono/mono/events","assignees_url":"https://api.github.com/repos/mono/mono/assignees{/user}","branches_url":"https://api.github.com/repos/mono/mono/branches{/branch}","tags_url":"https://api.github.com/repos/mono/mono/tags","blobs_url":"https://api.github.com/repos/mono/mono/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mono/mono/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mono/mono/git/refs{/sha}","trees_url":"https://api.github.com/repos/mono/mono/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mono/mono/statuses/{sha}","languages_url":"https://api.github.com/repos/mono/mono/languages","stargazers_url":"https://api.github.com/repos/mono/mono/stargazers","contributors_url":"https://api.github.com/repos/mono/mono/contributors","subscribers_url":"https://api.github.com/repos/mono/mono/subscribers","subscription_url":"https://api.github.com/repos/mono/mono/subscription","commits_url":"https://api.github.com/repos/mono/mono/commits{/sha}","git_commits_url":"https://api.github.com/repos/mono/mono/git/commits{/sha}","comments_url":"https://api.github.com/repos/mono/mono/comments{/number}","issue_comment_url":"https://api.github.com/repos/mono/mono/issues/comments{/number}","contents_url":"https://api.github.com/repos/mono/mono/contents/{+path}","compare_url":"https://api.github.com/repos/mono/mono/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mono/mono/merges","archive_url":"https://api.github.com/repos/mono/mono/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mono/mono/downloads","issues_url":"https://api.github.com/repos/mono/mono/issues{/number}","pulls_url":"https://api.github.com/repos/mono/mono/pulls{/number}","milestones_url":"https://api.github.com/repos/mono/mono/milestones{/number}","notifications_url":"https://api.github.com/repos/mono/mono/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mono/mono/labels{/name}","releases_url":"https://api.github.com/repos/mono/mono/releases{/id}","created_at":"2010-07-22T18:06:36Z","updated_at":"2015-12-18T17:23:17Z","pushed_at":"2015-12-18T18:31:37Z","git_url":"git://github.com/mono/mono.git","ssh_url":"git@github.com:mono/mono.git","clone_url":"https://github.com/mono/mono.git","svn_url":"https://github.com/mono/mono","homepage":"http://www.mono-project.com/","size":480773,"stargazers_count":4130,"watchers_count":4130,"language":"C#","has_issues":false,"has_downloads":false,"has_wiki":false,"has_pages":false,"forks_count":2035,"mirror_url":null,"open_issues_count":123,"forks":2035,"open_issues":123,"watchers":4130,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mono/mono/pulls/2366"},"html":{"href":"https://github.com/mono/mono/pull/2366"},"issue":{"href":"https://api.github.com/repos/mono/mono/issues/2366"},"comments":{"href":"https://api.github.com/repos/mono/mono/issues/2366/comments"},"review_comments":{"href":"https://api.github.com/repos/mono/mono/pulls/2366/comments"},"review_comment":{"href":"https://api.github.com/repos/mono/mono/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mono/mono/pulls/2366/commits"},"statuses":{"href":"https://api.github.com/repos/mono/mono/statuses/1dc0598badaed161efa92fd261de7e555d1a2b13"}}}}