{"action":"created","comment":{"url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/comments/48116377","id":48116377,"diff_hunk":"@@ -0,0 +1,28 @@\n+// RUN: %swift_exec %s -o %built_tests_dir/SingleFailingTestCase\n+// RUN: %built_tests_dir/SingleFailingTestCase > %test_output || true\n+// RUN: %xctest-checker %test_output %s\n+// CHECK: Test Case 'SingleFailingTestCase.test_fails' started.\n+// CHECK: .*/Tests/Functional/SingleFailingTestCase/main.swift:24: error: SingleFailingTestCase.test_fails : \n+// CHECK: Test Case 'SingleFailingTestCase.test_fails' failed \\(\\d+\\.\\d+ seconds\\).\n+// CHECK: Executed 1 test, with 1 failure 0 unexpected\\) in \\d+\\.\\d+ \\(\\d+\\.\\d+\\) seconds\n+// CHECK: Total executed 1 test, with 1 failure \\(0 unexpected\\) in \\d+\\.\\d+ \\(\\d+\\.\\d+\\) seconds","path":"Tests/Functional/SingleFailingTestCase/main.swift","position":8,"original_position":8,"commit_id":"f352a8363da2ca872608a44e6e1bdc1cdfaf40a7","original_commit_id":"f352a8363da2ca872608a44e6e1bdc1cdfaf40a7","user":{"login":"modocache","id":552921,"avatar_url":"https://avatars.githubusercontent.com/u/552921?v=3","gravatar_id":"","url":"https://api.github.com/users/modocache","html_url":"https://github.com/modocache","followers_url":"https://api.github.com/users/modocache/followers","following_url":"https://api.github.com/users/modocache/following{/other_user}","gists_url":"https://api.github.com/users/modocache/gists{/gist_id}","starred_url":"https://api.github.com/users/modocache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/modocache/subscriptions","organizations_url":"https://api.github.com/users/modocache/orgs","repos_url":"https://api.github.com/users/modocache/repos","events_url":"https://api.github.com/users/modocache/events{/privacy}","received_events_url":"https://api.github.com/users/modocache/received_events","type":"User","site_admin":false},"body":"`FileCheck` uses a special syntax for regular expressions. `xctest_checker` parses everything as a regular expression. As a result, developers need to take extra care to escape certain characters when writing their checks.\r\n\r\nThis is suboptimal, and could be improved in the future, but `xctest_checker` remains a simple program as a result of this decision.","created_at":"2015-12-21T04:26:55Z","updated_at":"2015-12-21T04:26:55Z","html_url":"https://github.com/apple/swift-corelibs-xctest/pull/20#discussion_r48116377","pull_request_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20","_links":{"self":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/comments/48116377"},"html":{"href":"https://github.com/apple/swift-corelibs-xctest/pull/20#discussion_r48116377"},"pull_request":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20"}}},"pull_request":{"url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20","id":54219505,"html_url":"https://github.com/apple/swift-corelibs-xctest/pull/20","diff_url":"https://github.com/apple/swift-corelibs-xctest/pull/20.diff","patch_url":"https://github.com/apple/swift-corelibs-xctest/pull/20.patch","issue_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/20","number":20,"state":"open","locked":false,"title":"[Tests] Add functional tests ","user":{"login":"modocache","id":552921,"avatar_url":"https://avatars.githubusercontent.com/u/552921?v=3","gravatar_id":"","url":"https://api.github.com/users/modocache","html_url":"https://github.com/modocache","followers_url":"https://api.github.com/users/modocache/followers","following_url":"https://api.github.com/users/modocache/following{/other_user}","gists_url":"https://api.github.com/users/modocache/gists{/gist_id}","starred_url":"https://api.github.com/users/modocache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/modocache/subscriptions","organizations_url":"https://api.github.com/users/modocache/orgs","repos_url":"https://api.github.com/users/modocache/repos","events_url":"https://api.github.com/users/modocache/events{/privacy}","received_events_url":"https://api.github.com/users/modocache/received_events","type":"User","site_admin":false},"body":"## What's in this pull request?\r\n\r\nYet another attempt at #10, this pull request adds functional tests that verify the output of programs that use XCTest. This pull request is based off of the discussion on #14, and uses `lit` as a test runner.\r\n\r\nA developer can kick off the tests by either running `build_script.py --test` or by building and running the `SwiftXCTestFunctionalTests` Xcode target.\r\n\r\nHere's a breakdown of what happens when a user kicks off the tests:\r\n\r\n1. `../llvm/utils/lit/lit.py Tests/Functional` is run, which kicks off the configuration in `Tests/Functional/lit.cfg`.\r\n2. `lit.cfg` uses environment variables to provide substitutions, such as `%swiftc`, to the tests. On Linux, these environment variables are set as part of `build_script.py`. On OS X, these environment variables are provided by `xcodebuild`.\r\n3. `lit` finds every Swift source file that contains a `RUN:` command and executes those commands.\r\n4. The `RUN:` commands in `SingleFailingTestCase/main.swift` compiles the source into an executable in a temporary directory, runs that executable, and checks the output. Output verification is done via `xctest_checker`, a Python script.\r\n5. The checker raises an exception and produces a failing exit code for the test runner if any tests fail.\r\n\r\n## Why merge this pull request?\r\n\r\n- It adds a functional test suite to this project. These tests will allow us to prevent regressions and verify the behavior of XCTest. For example, I noticed the typo in #18 as a result of functional testing.\r\n- Adding tests is as easy as adding a single file: `Tests/Functional/MyNewTest/main.swift`. Unlike #10 and #14, new tests don't need to be configured in the Xcode project to work.\r\n\r\n## What downsides are there to merging this pull request?\r\n\r\n- The test runner, because it uses `lit`, is dependent upon the `llvm` codebase being installed in `../llvm`. Alternatively, it could check for `pip` or `easy_install` being installed on the host system, and if available use them to install https://pypi.python.org/pypi/lit. Either way, the test suite is not completely independent: it relies on either `llvm` being cloned ahead of time, or on an Internet connection to install `lit` from the Python Package Index.\r\n- The test runner in #10 could verify the exit code of the program. `lit` is unable to do so. The tests verify the output, but not whether the exit code emitted by XCTest indicated a failure. @ddunbar, is it possible to test this using `lit`?\r\n- Functional tests only test the system as a whole. Ideally, these would be complemented with a unit test suite that could verify the behavior of smaller components in XCTest. I'd certainly like to pursue this idea in the future, but I think these tests are beneficial in their own way. They're not meant to be the only tests for this project.","created_at":"2015-12-21T04:19:58Z","updated_at":"2015-12-21T04:26:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2aa2791e5baaf99dd9851a843f114a9e6e111711","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20/commits","review_comments_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20/comments","review_comment_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/20/comments","statuses_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/statuses/f352a8363da2ca872608a44e6e1bdc1cdfaf40a7","head":{"label":"modocache:lit-tests-v2","ref":"lit-tests-v2","sha":"f352a8363da2ca872608a44e6e1bdc1cdfaf40a7","user":{"login":"modocache","id":552921,"avatar_url":"https://avatars.githubusercontent.com/u/552921?v=3","gravatar_id":"","url":"https://api.github.com/users/modocache","html_url":"https://github.com/modocache","followers_url":"https://api.github.com/users/modocache/followers","following_url":"https://api.github.com/users/modocache/following{/other_user}","gists_url":"https://api.github.com/users/modocache/gists{/gist_id}","starred_url":"https://api.github.com/users/modocache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/modocache/subscriptions","organizations_url":"https://api.github.com/users/modocache/orgs","repos_url":"https://api.github.com/users/modocache/repos","events_url":"https://api.github.com/users/modocache/events{/privacy}","received_events_url":"https://api.github.com/users/modocache/received_events","type":"User","site_admin":false},"repo":{"id":47346102,"name":"swift-corelibs-xctest","full_name":"modocache/swift-corelibs-xctest","owner":{"login":"modocache","id":552921,"avatar_url":"https://avatars.githubusercontent.com/u/552921?v=3","gravatar_id":"","url":"https://api.github.com/users/modocache","html_url":"https://github.com/modocache","followers_url":"https://api.github.com/users/modocache/followers","following_url":"https://api.github.com/users/modocache/following{/other_user}","gists_url":"https://api.github.com/users/modocache/gists{/gist_id}","starred_url":"https://api.github.com/users/modocache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/modocache/subscriptions","organizations_url":"https://api.github.com/users/modocache/orgs","repos_url":"https://api.github.com/users/modocache/repos","events_url":"https://api.github.com/users/modocache/events{/privacy}","received_events_url":"https://api.github.com/users/modocache/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/modocache/swift-corelibs-xctest","description":"The XCTest Project, A Swift core library for providing unit test support","fork":true,"url":"https://api.github.com/repos/modocache/swift-corelibs-xctest","forks_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/forks","keys_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/keys{/key_id}","collaborators_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/teams","hooks_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/hooks","issue_events_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/issues/events{/number}","events_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/events","assignees_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/assignees{/user}","branches_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/branches{/branch}","tags_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/tags","blobs_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/git/refs{/sha}","trees_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/git/trees{/sha}","statuses_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/statuses/{sha}","languages_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/languages","stargazers_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/stargazers","contributors_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/contributors","subscribers_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/subscribers","subscription_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/subscription","commits_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/commits{/sha}","git_commits_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/git/commits{/sha}","comments_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/comments{/number}","issue_comment_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/issues/comments{/number}","contents_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/contents/{+path}","compare_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/compare/{base}...{head}","merges_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/merges","archive_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/downloads","issues_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/issues{/number}","pulls_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/pulls{/number}","milestones_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/milestones{/number}","notifications_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/labels{/name}","releases_url":"https://api.github.com/repos/modocache/swift-corelibs-xctest/releases{/id}","created_at":"2015-12-03T16:50:59Z","updated_at":"2015-12-03T22:38:18Z","pushed_at":"2015-12-21T04:04:17Z","git_url":"git://github.com/modocache/swift-corelibs-xctest.git","ssh_url":"git@github.com:modocache/swift-corelibs-xctest.git","clone_url":"https://github.com/modocache/swift-corelibs-xctest.git","svn_url":"https://github.com/modocache/swift-corelibs-xctest","homepage":"swift.org","size":62,"stargazers_count":0,"watchers_count":0,"language":"Swift","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apple:master","ref":"master","sha":"a30ae40556e7b110d2e53a8909fc2e4ba00cccb5","user":{"login":"apple","id":10639145,"avatar_url":"https://avatars.githubusercontent.com/u/10639145?v=3","gravatar_id":"","url":"https://api.github.com/users/apple","html_url":"https://github.com/apple","followers_url":"https://api.github.com/users/apple/followers","following_url":"https://api.github.com/users/apple/following{/other_user}","gists_url":"https://api.github.com/users/apple/gists{/gist_id}","starred_url":"https://api.github.com/users/apple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apple/subscriptions","organizations_url":"https://api.github.com/users/apple/orgs","repos_url":"https://api.github.com/users/apple/repos","events_url":"https://api.github.com/users/apple/events{/privacy}","received_events_url":"https://api.github.com/users/apple/received_events","type":"Organization","site_admin":false},"repo":{"id":45864027,"name":"swift-corelibs-xctest","full_name":"apple/swift-corelibs-xctest","owner":{"login":"apple","id":10639145,"avatar_url":"https://avatars.githubusercontent.com/u/10639145?v=3","gravatar_id":"","url":"https://api.github.com/users/apple","html_url":"https://github.com/apple","followers_url":"https://api.github.com/users/apple/followers","following_url":"https://api.github.com/users/apple/following{/other_user}","gists_url":"https://api.github.com/users/apple/gists{/gist_id}","starred_url":"https://api.github.com/users/apple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apple/subscriptions","organizations_url":"https://api.github.com/users/apple/orgs","repos_url":"https://api.github.com/users/apple/repos","events_url":"https://api.github.com/users/apple/events{/privacy}","received_events_url":"https://api.github.com/users/apple/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apple/swift-corelibs-xctest","description":"The XCTest Project, A Swift core library for providing unit test support","fork":false,"url":"https://api.github.com/repos/apple/swift-corelibs-xctest","forks_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/forks","keys_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/teams","hooks_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/hooks","issue_events_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/events{/number}","events_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/events","assignees_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/assignees{/user}","branches_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/branches{/branch}","tags_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/tags","blobs_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/git/refs{/sha}","trees_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/statuses/{sha}","languages_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/languages","stargazers_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/stargazers","contributors_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/contributors","subscribers_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/subscribers","subscription_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/subscription","commits_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/commits{/sha}","git_commits_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/git/commits{/sha}","comments_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/comments{/number}","issue_comment_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/comments{/number}","contents_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/contents/{+path}","compare_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/merges","archive_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/downloads","issues_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues{/number}","pulls_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls{/number}","milestones_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/milestones{/number}","notifications_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/labels{/name}","releases_url":"https://api.github.com/repos/apple/swift-corelibs-xctest/releases{/id}","created_at":"2015-11-09T20:15:01Z","updated_at":"2015-12-20T11:10:45Z","pushed_at":"2015-12-21T04:19:59Z","git_url":"git://github.com/apple/swift-corelibs-xctest.git","ssh_url":"git@github.com:apple/swift-corelibs-xctest.git","clone_url":"https://github.com/apple/swift-corelibs-xctest.git","svn_url":"https://github.com/apple/swift-corelibs-xctest","homepage":"swift.org","size":64,"stargazers_count":374,"watchers_count":374,"language":"Swift","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":80,"mirror_url":null,"open_issues_count":3,"forks":80,"open_issues":3,"watchers":374,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20"},"html":{"href":"https://github.com/apple/swift-corelibs-xctest/pull/20"},"issue":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/20"},"comments":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/issues/20/comments"},"review_comments":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20/comments"},"review_comment":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/pulls/20/commits"},"statuses":{"href":"https://api.github.com/repos/apple/swift-corelibs-xctest/statuses/f352a8363da2ca872608a44e6e1bdc1cdfaf40a7"}}}}