{"action":"created","issue":{"url":"https://api.github.com/repos/atom/atom/issues/8968","labels_url":"https://api.github.com/repos/atom/atom/issues/8968/labels{/name}","comments_url":"https://api.github.com/repos/atom/atom/issues/8968/comments","events_url":"https://api.github.com/repos/atom/atom/issues/8968/events","html_url":"https://github.com/atom/atom/pull/8968","id":108586860,"number":8968,"title":"Enable custom package testing environments","user":{"login":"nathansobo","id":1789,"avatar_url":"https://avatars.githubusercontent.com/u/1789?v=3","gravatar_id":"","url":"https://api.github.com/users/nathansobo","html_url":"https://github.com/nathansobo","followers_url":"https://api.github.com/users/nathansobo/followers","following_url":"https://api.github.com/users/nathansobo/following{/other_user}","gists_url":"https://api.github.com/users/nathansobo/gists{/gist_id}","starred_url":"https://api.github.com/users/nathansobo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathansobo/subscriptions","organizations_url":"https://api.github.com/users/nathansobo/orgs","repos_url":"https://api.github.com/users/nathansobo/repos","events_url":"https://api.github.com/users/nathansobo/events{/privacy}","received_events_url":"https://api.github.com/users/nathansobo/received_events","type":"User","site_admin":true},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2015-09-28T04:15:42Z","updated_at":"2015-10-15T04:23:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/atom/atom/pulls/8968","html_url":"https://github.com/atom/atom/pull/8968","diff_url":"https://github.com/atom/atom/pull/8968.diff","patch_url":"https://github.com/atom/atom/pull/8968.patch"},"body":"This PR aims to improve the experience of running tests in Atom in a few ways.\r\n\r\n* Package authors will be able to specify an `atomTestRunner` field in their `package.json` containing a require path. If the field is present, Atom will resolve the path relative to the package's root, expecting the required module to export a function which it will invoke with an object containing the following parameters.\r\n  * `testPaths` An array of paths to tests to run. Could be paths to files or directories.\r\n  * `buildAtomEnvironment` A function that can be called to construct an instance of the `atom` global (it won't automatically be assigned to `global.atom`, that is left to the test runner to decide.\r\n  * `logFile` An optional path to a log file to which test output should be logged.\r\n  * `headless` A boolean indicating whether or not the tests are running headless.\r\n  * `legacyTestRunner` This function can be invoked to run the legacy Jasmine runner, giving packages a chance to transition to a new test runner while maintaining a subset of their tests in the old environment.\r\n* The test runner function will be expected to return a promise that resolves with an exit code. When tests are being run in headless mode Atom will exit with this code when the promise resolves. In interactive mode the promise will be ignored.\r\n* Running tests via the `atom` CLI has been streamlined. Simply run `atom --test path1 path2` to invoke the test runner on the listed paths, expanded relative to the working directory. The `--spec-directory` option will be deprecated.\r\n* The `buildAtomEnvironment` function will construct a clean Atom environment whose state can be reset via a call to `reset` in between test runs.\r\n\r\nRefs #8917\r\n/cc @bolinfest @maxbrunsfeld\r\n\r\n### Additional Thoughts\r\n\r\nAtom's current testing environment has been around since the very first day of Atom development, and is quite a mess. We assign all kinds of globals and mock the clock in evil and inconsistent ways that can cause quite a few headaches when writing tests for new packages.\r\n\r\nThis PR will give us a good way to make a fresh start. Ideally, a default test runner isn't something we should be supporting in Atom core. Every package should specify its preferred runner, which it can pull in as a development dependency and which Atom can interact with via an evolving interface. It would be good to have a recommended test framework. As it stands I'd probably want to switch bundled packages over to Mocha/Chai, unless people have other suggestions. Someday in the distant future we can finally delete the tangled mess that is our current default testing environment, but we'll keep it around for now.\r\n\r\n### Tasks\r\n\r\n* [x] Basic clean up around test window initialization\r\n* [x] Support specifying multiple test paths as arguments from the CLI.\r\n* [x] Support `atomTestRunner` field containing a path.\r\n* [x] Support multiple test paths in the default jasmine test runner.\r\n* [x] Clean up Atom environment construction so we can instantiate the Atom environment without assigning to globals.\r\n* [x] Pass `buildAtomEnvironment` to test runners and use it to assign the `atom` global in the default test runner, rather than in the test window initialization script.\r\n* [x] Rename `Atom` to `AtomEnvironment`\r\n* [x] Add a `reset` method to Atom environment\r\n* [x] Pass `legacyTestRunner` parameter.\r\n* [x] Fix docs generation.\r\n* [x] Drop the `--spec-directory` CLI option\r\n  * [ ] Stop using `--spec-directory` in `apm test` (https://github.com/atom/apm/pull/445)\r\n  * [x] Stop using `specDirectory` parameter when running tests interactively\r\n* [x] Drop the `Run All Tests` command from Atom since it's broken.\r\n* [x] Shim `process.stdout.write` and `process.stderr.write` in the test window\r\n* [x] Support a timeout flag\r\n* [x] Expect the test runner to return a promise\r\n* [x] Update packages to not use deprecated methods\r\n  * [x] https://github.com/atom/welcome/pull/39\r\n  * [x] https://github.com/atom/markdown-preview/pull/316\r\n  * [x] https://github.com/atom/language-php/pull/105\r\n  * [x] https://github.com/atom/autocomplete-plus/pull/592\r\n  * [x] https://github.com/atom/snippets/pull/176\r\n  * [x] https://github.com/atom/autosave/pull/49\r\n  * [x] https://github.com/atom/tabs/pull/214\r\n  * [x] https://github.com/atom/language-javascript/pull/239\r\n  * [x] https://github.com/atom/find-and-replace/pull/551\r\n  * [x] https://github.com/atom/language-c/pull/93\r\n  * [x] https://github.com/atom/find-and-replace/pull/552\r\n  * [x] https://github.com/atom/fuzzy-finder/pull/144\r\n* [x] Add user-facing documentation for the new workflow and options \r\n* [x] Move remaining uses of APIs that interact with main process such as `ipc` to methods on `ApplicationDelegate`\r\n* [x] Assign the `defaultTarget` of the keymaps in the `AtomEnvironment` constructor.\r\n* [x] Parameterize the `configDirPath` and don't attempt any disk I/O in `Config` or `AtomEnvironment` if it is undefined. This will allow us to remove several mocks from the spec helper that prevent reading and writing state.\r\n* [x] Move `config.load()` into the `AtomEnvironment` constructor, which will be a no-op in specs where the config dir is undefined. This will allow us to move `loadBaseStyleSheets` back into the constructor as well so they are available in the environment's default state.\r\n* [x] Move subscription to `core.autoHideMenuBar` into `AtomEnvironment` constructor.\r\n* [ ] Get build green.\r\n* [ ] Document environment parameters (`applicationDelegate`, `window`, `document`)"},"comment":{"url":"https://api.github.com/repos/atom/atom/issues/comments/148277016","html_url":"https://github.com/atom/atom/pull/8968#issuecomment-148277016","issue_url":"https://api.github.com/repos/atom/atom/issues/8968","id":148277016,"user":{"login":"nathansobo","id":1789,"avatar_url":"https://avatars.githubusercontent.com/u/1789?v=3","gravatar_id":"","url":"https://api.github.com/users/nathansobo","html_url":"https://github.com/nathansobo","followers_url":"https://api.github.com/users/nathansobo/followers","following_url":"https://api.github.com/users/nathansobo/following{/other_user}","gists_url":"https://api.github.com/users/nathansobo/gists{/gist_id}","starred_url":"https://api.github.com/users/nathansobo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathansobo/subscriptions","organizations_url":"https://api.github.com/users/nathansobo/orgs","repos_url":"https://api.github.com/users/nathansobo/repos","events_url":"https://api.github.com/users/nathansobo/events{/privacy}","received_events_url":"https://api.github.com/users/nathansobo/received_events","type":"User","site_admin":true},"created_at":"2015-10-15T04:23:20Z","updated_at":"2015-10-15T04:23:20Z","body":"Okay, everything is green but core specs but I can't see any output from CI or reproduce locally. Almost there though."}}