{"action":"created","comment":{"url":"https://api.github.com/repos/apple/swift-evolution/pulls/comments/47830668","id":47830668,"diff_hunk":"@@ -0,0 +1,194 @@\n+# Swift Testing\n+\n+* Authors:\n+  [Max Howell](https://github.com/mxcl),\n+  [Daniel Dunbar](https://github.com/ddunbar),\n+  [Mattt Thompson](https://github.com/mattt)\n+* Status: **Review**\n+* Review Manager: Rick Ballard\n+\n+## Introduction\n+\n+Testing is an essential part of modern software development.\n+Tight integration of testing into the Swift Package Manager\n+will help ensure a stable and reliable packaging ecosystem.\n+\n+## Proposed Solution\n+\n+We propose to extend our conventional package directory layout\n+to accomodate test modules.\n+Any subdirectory of the package root directory named \"Tests\"\n+or any subdirectory of an existing module directory named \"Tests\"\n+will comprise a test module.\n+For example:\n+\n+    Package\n+    ├── Sources\n+    │   └── Foo\n+    │       └──Foo.swift\n+    └── Tests\n+        └── Foo\n+            └── Test.swift\n+\n+Or:\n+\n+    Package\n+    └── Sources\n+        ├── Foo.swift\n+        └── Tests\n+            └── Test.swift\n+\n+Or, for simpler projects:\n+\n+    Package\n+    ├── Sources\n+    │   └── Foo.swift\n+    └── Tests\n+        └── TestFoo.swift\n+        \n+The filename: `TestFoo.swift` is arbitrary.\n+\n+In the examples above\n+a test case is created for the module `Foo`\n+based on the sources in the relevant subdirectories.\n+\n+Additionally we will support directories called `FooTests`.\n+This layout style is prevalent in existing open source projects\n+and supporting it will minimize vexation for their authors.\n+However in the interest of consistency and the corresponding \n+reduction of cognitive-load when examining new Swift packages\n+we will not recommend this layout. For example:\n+\n+    Package\n+    └── Sources\n+    │   └── Foo.swift\n+    └── FooTests\n+        └── Test.swift\n+\n+Additionally, we propose that building a module\n+also builds that module's corresponding tests.\n+Although this would result in slightly increased build times,\n+we believe that tests are important enough to justify this\n+(one might even consider slow building tests to be a code smell).\n+We would prefer to go even further by executing the tests\n+each time a module is built as well,\n+but we understand that this would impede debug cycles.\n+\n+As an exception, when packages are built in release mode we will\n+not build tests because for release builds we should not enable\n+testability. However, considering the need for release-mode testing\n+this will be a future direction.\n+\n+### Command-Line Interface\n+\n+We propose the following syntax to execute tests:\n+\n+    $ swift build --test\n+\n+Or:\n+\n+    $ swift build -t\n+\n+In the future, we may choose to promote the `--test` option\n+to be a subcommand of the `swift` command itself:\n+\n+    $ swift test\n+\n+However, any such decision would warrant extensive design consideration,\n+so as to avoid polluting or crowding the command-line interface.\n+Should there be sufficient demand and justification for it, though,\n+it would be straightforward to add this functionality.\n+\n+### Command Output\n+\n+Executing a test from the terminal will produce user-readable output.\n+This should incorporate colorization and other formatting\n+similar to other testing tools\n+to indicate the success and failure of different tests.\n+For example:\n+\n+    $ swift test --output module\n+    Running tests for PackageX (x/100)\n+    .........x.....x...................\n+\n+    Completed\n+    Elapsed time: 0.2s\n+\n+    98 Success\n+     2 Failure\n+     1 Warning\n+\n+    FAILURE: Tests/TestsA.swift:24 testFoo()\n+    XCTAssertTrue expected true, got false\n+\n+    FAILURE: Tests/TestsB.swift:10 testBar()\n+    XCTAssertEqual\n+\n+    WARNING: Tests/TestsC.swift:1\n+    \"Some Warning\"\n+\n+An additional option may be passed to the testing command\n+to output JUnit-style XML or other formats that can be integrated\n+with continuous integration (CI) and other systems.\n+\n+### Backwards Compatibility\n+\n+In order to accomodate existing packages,\n+we will allow test module targets and their targets\n+to be overridden in the `Package.swift` manifest file.\n+However, this functionality will likely not be implemented\n+in the initial release of this feature,\n+and instead be added at a later point in time.\n+\n+### Automatic Dependency Determination\n+\n+Testing is important and it is important to make the barrier to testing\n+as minimal as possible. Thus, by analyzing the names of test targets,\n+we will automatically determine the most likely dependency of that test\n+and accomodate accordingly.\n+For example,\n+a test for \"Foo\" will depend on compilation of the library target `Foo`.\n+Any additional dependencies or dependencies that could not be automatically determined\n+would need to be specified in a package manifest separately.\n+\n+### Debug / Release Configuration\n+\n+Although tests built in debug configuration\n+are generally run against modules also build in debug configuration,\n+it is sometimes necessary to specify the build configuration for tests separately.\n+It is also sometimes necessary to explicitly specify this information for every build,\n+such as when building in a release configuration to execute performance tests.\n+We would like to eventually support these use cases,\n+however this will not be present in the initial implementation of this feature.\n+\n+### Testability\n+\n+Swift can build modules with \"testability\",\n+which allows tests to access entities with `internal` access control.\n+Because it would be tedious for users to specify this requirement for tests,\n+we intend to build debug builds with testability by default.\n+\n+### Test Frameworks\n+\n+Initially,\n+the Swift Package Manager will use `XCTest` as its underlying test framework.\n+\n+However, testing is an evolving artform,\n+so we'd like to support other approaches\n+that might allow frameworks other than XCTest\n+to be supported by the package manager.\n+We expect that such an implementation would take the form of\n+a Swift protocol that the package manager defines,\n+which other testing frameworks can adopt.","path":"proposals/0008-package-manager-testing.md","position":198,"original_position":182,"commit_id":"9aa76cc5aa9ccc63d126327f36f18256a216299f","original_commit_id":"ca7c2af34961588f49fc1cb7a20e7b8430895772","user":{"login":"bppr","id":394714,"avatar_url":"https://avatars.githubusercontent.com/u/394714?v=3","gravatar_id":"","url":"https://api.github.com/users/bppr","html_url":"https://github.com/bppr","followers_url":"https://api.github.com/users/bppr/followers","following_url":"https://api.github.com/users/bppr/following{/other_user}","gists_url":"https://api.github.com/users/bppr/gists{/gist_id}","starred_url":"https://api.github.com/users/bppr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bppr/subscriptions","organizations_url":"https://api.github.com/users/bppr/orgs","repos_url":"https://api.github.com/users/bppr/repos","events_url":"https://api.github.com/users/bppr/events{/privacy}","received_events_url":"https://api.github.com/users/bppr/received_events","type":"User","site_admin":false},"body":"^^ Agreed, but I think as long as that protocol isn't part of `XCTest` and is more of a part of the build-tool / package management infrastructure itself, my testing framework can adopt that protocol fairly easily, while XCTest can be integrated as a sensible default, but each package has a measure of control over what testing tools they want to use.\r\n\r\nAs someone developing a third-party testing tool, I really just don't want my testing framework to have an XCTest dependency _just_ so I can package it with the most common tools. As long as that happens, I'll be a happy camper.","created_at":"2015-12-16T20:40:57Z","updated_at":"2015-12-16T20:40:57Z","html_url":"https://github.com/apple/swift-evolution/pull/51#discussion_r47830668","pull_request_url":"https://api.github.com/repos/apple/swift-evolution/pulls/51","_links":{"self":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/comments/47830668"},"html":{"href":"https://github.com/apple/swift-evolution/pull/51#discussion_r47830668"},"pull_request":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/51"}}},"pull_request":{"url":"https://api.github.com/repos/apple/swift-evolution/pulls/51","id":53459864,"html_url":"https://github.com/apple/swift-evolution/pull/51","diff_url":"https://github.com/apple/swift-evolution/pull/51.diff","patch_url":"https://github.com/apple/swift-evolution/pull/51.patch","issue_url":"https://api.github.com/repos/apple/swift-evolution/issues/51","number":51,"state":"open","locked":false,"title":"Proposal for testing support in SwiftPM","user":{"login":"mxcl","id":58962,"avatar_url":"https://avatars.githubusercontent.com/u/58962?v=3","gravatar_id":"","url":"https://api.github.com/users/mxcl","html_url":"https://github.com/mxcl","followers_url":"https://api.github.com/users/mxcl/followers","following_url":"https://api.github.com/users/mxcl/following{/other_user}","gists_url":"https://api.github.com/users/mxcl/gists{/gist_id}","starred_url":"https://api.github.com/users/mxcl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mxcl/subscriptions","organizations_url":"https://api.github.com/users/mxcl/orgs","repos_url":"https://api.github.com/users/mxcl/repos","events_url":"https://api.github.com/users/mxcl/events{/privacy}","received_events_url":"https://api.github.com/users/mxcl/received_events","type":"User","site_admin":false},"body":"Our proposal for how testing infrastructure should operate and be implemented in the package manager.","created_at":"2015-12-12T00:04:41Z","updated_at":"2015-12-16T20:40:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0dacb16d8bee369f3c8258fd955db1d055ac9e7d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apple/swift-evolution/pulls/51/commits","review_comments_url":"https://api.github.com/repos/apple/swift-evolution/pulls/51/comments","review_comment_url":"https://api.github.com/repos/apple/swift-evolution/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apple/swift-evolution/issues/51/comments","statuses_url":"https://api.github.com/repos/apple/swift-evolution/statuses/9aa76cc5aa9ccc63d126327f36f18256a216299f","head":{"label":"mxcl:swiftpm-testing","ref":"swiftpm-testing","sha":"9aa76cc5aa9ccc63d126327f36f18256a216299f","user":{"login":"mxcl","id":58962,"avatar_url":"https://avatars.githubusercontent.com/u/58962?v=3","gravatar_id":"","url":"https://api.github.com/users/mxcl","html_url":"https://github.com/mxcl","followers_url":"https://api.github.com/users/mxcl/followers","following_url":"https://api.github.com/users/mxcl/following{/other_user}","gists_url":"https://api.github.com/users/mxcl/gists{/gist_id}","starred_url":"https://api.github.com/users/mxcl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mxcl/subscriptions","organizations_url":"https://api.github.com/users/mxcl/orgs","repos_url":"https://api.github.com/users/mxcl/repos","events_url":"https://api.github.com/users/mxcl/events{/privacy}","received_events_url":"https://api.github.com/users/mxcl/received_events","type":"User","site_admin":false},"repo":{"id":47857026,"name":"swift-evolution","full_name":"mxcl/swift-evolution","owner":{"login":"mxcl","id":58962,"avatar_url":"https://avatars.githubusercontent.com/u/58962?v=3","gravatar_id":"","url":"https://api.github.com/users/mxcl","html_url":"https://github.com/mxcl","followers_url":"https://api.github.com/users/mxcl/followers","following_url":"https://api.github.com/users/mxcl/following{/other_user}","gists_url":"https://api.github.com/users/mxcl/gists{/gist_id}","starred_url":"https://api.github.com/users/mxcl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mxcl/subscriptions","organizations_url":"https://api.github.com/users/mxcl/orgs","repos_url":"https://api.github.com/users/mxcl/repos","events_url":"https://api.github.com/users/mxcl/events{/privacy}","received_events_url":"https://api.github.com/users/mxcl/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mxcl/swift-evolution","description":"This maintains proposals for changes and user-visible enhancements to the Swift Programming Language.","fork":true,"url":"https://api.github.com/repos/mxcl/swift-evolution","forks_url":"https://api.github.com/repos/mxcl/swift-evolution/forks","keys_url":"https://api.github.com/repos/mxcl/swift-evolution/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mxcl/swift-evolution/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mxcl/swift-evolution/teams","hooks_url":"https://api.github.com/repos/mxcl/swift-evolution/hooks","issue_events_url":"https://api.github.com/repos/mxcl/swift-evolution/issues/events{/number}","events_url":"https://api.github.com/repos/mxcl/swift-evolution/events","assignees_url":"https://api.github.com/repos/mxcl/swift-evolution/assignees{/user}","branches_url":"https://api.github.com/repos/mxcl/swift-evolution/branches{/branch}","tags_url":"https://api.github.com/repos/mxcl/swift-evolution/tags","blobs_url":"https://api.github.com/repos/mxcl/swift-evolution/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mxcl/swift-evolution/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mxcl/swift-evolution/git/refs{/sha}","trees_url":"https://api.github.com/repos/mxcl/swift-evolution/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mxcl/swift-evolution/statuses/{sha}","languages_url":"https://api.github.com/repos/mxcl/swift-evolution/languages","stargazers_url":"https://api.github.com/repos/mxcl/swift-evolution/stargazers","contributors_url":"https://api.github.com/repos/mxcl/swift-evolution/contributors","subscribers_url":"https://api.github.com/repos/mxcl/swift-evolution/subscribers","subscription_url":"https://api.github.com/repos/mxcl/swift-evolution/subscription","commits_url":"https://api.github.com/repos/mxcl/swift-evolution/commits{/sha}","git_commits_url":"https://api.github.com/repos/mxcl/swift-evolution/git/commits{/sha}","comments_url":"https://api.github.com/repos/mxcl/swift-evolution/comments{/number}","issue_comment_url":"https://api.github.com/repos/mxcl/swift-evolution/issues/comments{/number}","contents_url":"https://api.github.com/repos/mxcl/swift-evolution/contents/{+path}","compare_url":"https://api.github.com/repos/mxcl/swift-evolution/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mxcl/swift-evolution/merges","archive_url":"https://api.github.com/repos/mxcl/swift-evolution/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mxcl/swift-evolution/downloads","issues_url":"https://api.github.com/repos/mxcl/swift-evolution/issues{/number}","pulls_url":"https://api.github.com/repos/mxcl/swift-evolution/pulls{/number}","milestones_url":"https://api.github.com/repos/mxcl/swift-evolution/milestones{/number}","notifications_url":"https://api.github.com/repos/mxcl/swift-evolution/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mxcl/swift-evolution/labels{/name}","releases_url":"https://api.github.com/repos/mxcl/swift-evolution/releases{/id}","created_at":"2015-12-12T00:01:44Z","updated_at":"2015-12-11T22:16:27Z","pushed_at":"2015-12-16T19:37:12Z","git_url":"git://github.com/mxcl/swift-evolution.git","ssh_url":"git@github.com:mxcl/swift-evolution.git","clone_url":"https://github.com/mxcl/swift-evolution.git","svn_url":"https://github.com/mxcl/swift-evolution","homepage":"","size":91,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apple:master","ref":"master","sha":"d220ae4d6a272e70134d9ee39345d032515a4e00","user":{"login":"apple","id":10639145,"avatar_url":"https://avatars.githubusercontent.com/u/10639145?v=3","gravatar_id":"","url":"https://api.github.com/users/apple","html_url":"https://github.com/apple","followers_url":"https://api.github.com/users/apple/followers","following_url":"https://api.github.com/users/apple/following{/other_user}","gists_url":"https://api.github.com/users/apple/gists{/gist_id}","starred_url":"https://api.github.com/users/apple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apple/subscriptions","organizations_url":"https://api.github.com/users/apple/orgs","repos_url":"https://api.github.com/users/apple/repos","events_url":"https://api.github.com/users/apple/events{/privacy}","received_events_url":"https://api.github.com/users/apple/received_events","type":"Organization","site_admin":false},"repo":{"id":45497910,"name":"swift-evolution","full_name":"apple/swift-evolution","owner":{"login":"apple","id":10639145,"avatar_url":"https://avatars.githubusercontent.com/u/10639145?v=3","gravatar_id":"","url":"https://api.github.com/users/apple","html_url":"https://github.com/apple","followers_url":"https://api.github.com/users/apple/followers","following_url":"https://api.github.com/users/apple/following{/other_user}","gists_url":"https://api.github.com/users/apple/gists{/gist_id}","starred_url":"https://api.github.com/users/apple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apple/subscriptions","organizations_url":"https://api.github.com/users/apple/orgs","repos_url":"https://api.github.com/users/apple/repos","events_url":"https://api.github.com/users/apple/events{/privacy}","received_events_url":"https://api.github.com/users/apple/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apple/swift-evolution","description":"This maintains proposals for changes and user-visible enhancements to the Swift Programming Language.","fork":false,"url":"https://api.github.com/repos/apple/swift-evolution","forks_url":"https://api.github.com/repos/apple/swift-evolution/forks","keys_url":"https://api.github.com/repos/apple/swift-evolution/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apple/swift-evolution/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apple/swift-evolution/teams","hooks_url":"https://api.github.com/repos/apple/swift-evolution/hooks","issue_events_url":"https://api.github.com/repos/apple/swift-evolution/issues/events{/number}","events_url":"https://api.github.com/repos/apple/swift-evolution/events","assignees_url":"https://api.github.com/repos/apple/swift-evolution/assignees{/user}","branches_url":"https://api.github.com/repos/apple/swift-evolution/branches{/branch}","tags_url":"https://api.github.com/repos/apple/swift-evolution/tags","blobs_url":"https://api.github.com/repos/apple/swift-evolution/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apple/swift-evolution/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apple/swift-evolution/git/refs{/sha}","trees_url":"https://api.github.com/repos/apple/swift-evolution/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apple/swift-evolution/statuses/{sha}","languages_url":"https://api.github.com/repos/apple/swift-evolution/languages","stargazers_url":"https://api.github.com/repos/apple/swift-evolution/stargazers","contributors_url":"https://api.github.com/repos/apple/swift-evolution/contributors","subscribers_url":"https://api.github.com/repos/apple/swift-evolution/subscribers","subscription_url":"https://api.github.com/repos/apple/swift-evolution/subscription","commits_url":"https://api.github.com/repos/apple/swift-evolution/commits{/sha}","git_commits_url":"https://api.github.com/repos/apple/swift-evolution/git/commits{/sha}","comments_url":"https://api.github.com/repos/apple/swift-evolution/comments{/number}","issue_comment_url":"https://api.github.com/repos/apple/swift-evolution/issues/comments{/number}","contents_url":"https://api.github.com/repos/apple/swift-evolution/contents/{+path}","compare_url":"https://api.github.com/repos/apple/swift-evolution/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apple/swift-evolution/merges","archive_url":"https://api.github.com/repos/apple/swift-evolution/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apple/swift-evolution/downloads","issues_url":"https://api.github.com/repos/apple/swift-evolution/issues{/number}","pulls_url":"https://api.github.com/repos/apple/swift-evolution/pulls{/number}","milestones_url":"https://api.github.com/repos/apple/swift-evolution/milestones{/number}","notifications_url":"https://api.github.com/repos/apple/swift-evolution/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apple/swift-evolution/labels{/name}","releases_url":"https://api.github.com/repos/apple/swift-evolution/releases{/id}","created_at":"2015-11-03T21:49:37Z","updated_at":"2015-12-16T19:51:09Z","pushed_at":"2015-12-16T19:37:12Z","git_url":"git://github.com/apple/swift-evolution.git","ssh_url":"git@github.com:apple/swift-evolution.git","clone_url":"https://github.com/apple/swift-evolution.git","svn_url":"https://github.com/apple/swift-evolution","homepage":"","size":183,"stargazers_count":1983,"watchers_count":1983,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":207,"mirror_url":null,"open_issues_count":11,"forks":207,"open_issues":11,"watchers":1983,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/51"},"html":{"href":"https://github.com/apple/swift-evolution/pull/51"},"issue":{"href":"https://api.github.com/repos/apple/swift-evolution/issues/51"},"comments":{"href":"https://api.github.com/repos/apple/swift-evolution/issues/51/comments"},"review_comments":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/51/comments"},"review_comment":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apple/swift-evolution/pulls/51/commits"},"statuses":{"href":"https://api.github.com/repos/apple/swift-evolution/statuses/9aa76cc5aa9ccc63d126327f36f18256a216299f"}}}}