{"action":"created","issue":{"url":"https://api.github.com/repos/craftyjs/Crafty/issues/986","labels_url":"https://api.github.com/repos/craftyjs/Crafty/issues/986/labels{/name}","comments_url":"https://api.github.com/repos/craftyjs/Crafty/issues/986/comments","events_url":"https://api.github.com/repos/craftyjs/Crafty/issues/986/events","html_url":"https://github.com/craftyjs/Crafty/pull/986","id":119413757,"number":986,"title":"Add WebDriver tests","user":{"login":"mucaho","id":3935691,"avatar_url":"https://avatars.githubusercontent.com/u/3935691?v=3","gravatar_id":"","url":"https://api.github.com/users/mucaho","html_url":"https://github.com/mucaho","followers_url":"https://api.github.com/users/mucaho/followers","following_url":"https://api.github.com/users/mucaho/following{/other_user}","gists_url":"https://api.github.com/users/mucaho/gists{/gist_id}","starred_url":"https://api.github.com/users/mucaho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mucaho/subscriptions","organizations_url":"https://api.github.com/users/mucaho/orgs","repos_url":"https://api.github.com/users/mucaho/repos","events_url":"https://api.github.com/users/mucaho/events{/privacy}","received_events_url":"https://api.github.com/users/mucaho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/craftyjs/Crafty/labels/meta","name":"meta","color":"fad8c7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-30T03:43:54Z","updated_at":"2015-12-13T19:00:49Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/craftyjs/Crafty/pulls/986","html_url":"https://github.com/craftyjs/Crafty/pull/986","diff_url":"https://github.com/craftyjs/Crafty/pull/986.diff","patch_url":"https://github.com/craftyjs/Crafty/pull/986.patch"},"body":"## What are those tests?\r\n\r\nThese tests consist of two test files:\r\n* One test file runs in a browser (either a local PhantomJS instance, or on different browsers provided by Sauce Labs).\r\n\r\n* The other test file runs in node and actually \"drives\" the browser - opens urls, clicks on things in the window and takes screenshots. The latter (\"webdriver\") test file makes certain assertions, making the tests pass or fail (in our case we mostly compare screenshots).\r\n\r\n## Where do I insert new test cases?\r\n\r\n* Look at `tests/webdriver/index-webdriver.html`. Insert the test case name at the start of the file (you will see comment). Optionally specify a condition when it should be executed - you don't want to run WebGL tests in old browsers.\r\n\r\n* Create two test files, one named `tests/webdriver/<my-test-name>.html` and the other one named `tests/webdriver/<my-test-name>.js` (the former being the code that will run in browser, the latter being the webdriver code). Copy & paste existing test files in order to ensure a base structure, that the test internals rely on.\r\n\r\n* When you first run your new tests via `npm test` (== `grunt check`), or more specifically via `grunt test-local-webdriver`, you will probably see error messages complaining about non-existent screenshots. This works as intended: The test internals look for corresponding screenshots in `tests/assets/webdriver/<my-test-name>-expected.png`. If they can't find them, they will be regenerated and error thrown. A 2nd test run should give no more errors. Deleting existing expected screenshots is also possible. The expected screenshots are committed to the repository.\r\n* Failed screenshots from local tests can be observed in the `build/webdriver/failed` directory. Failed screenshots from SauceLabs tests can be observed on [Crafty Distro#regression-tests](https://github.com/craftyjs/Crafty-Distro/tree/regression-tests).\r\n\r\n## Anything to keep in mind when constructing a browser test page?\r\n\r\n* Look at existing test pages.\r\n\r\n* Preserve the base structure given by other test pages. Crafty will be initted for you with a resolution of 320 x 240 px, in the default `cr-stage` div. Background will be set to gray. CSS rules will remove all margins and paddings.\r\n\r\n* This means all elements in test page should fit into the given resolution! If you want more freedom, checkout `template-generic` and `template-local`.\r\n\r\n## Ok, I constructed a browser test page. Which webdriver test commands can I use?\r\n\r\n* Look at existing test cases. Don't forget the `QUnit.module` preamble.\r\n\r\n* There is a global `browser` instance which you send commands to. The commands are all asynchronous and chain together like [promises](https://github.com/kriskowal/q) - don't forget to `return` the promise chain! All available commands are listed in [WebdriverIO API docs](http://webdriver.io/api.html).\r\n\r\n* Additional commands are added by the test internals, which should be preferred to the WebdriverIO ones - they are specifically tailored to the (sometimes extreme) differences to features available in different browser drivers:\r\n   * `testUrl` - navigates to proper url matching your `<my-test-name>`\r\n   * `assertResemble` - compares newly taken screenshots to expected ones, or records new expected screenshots (as explained above)\r\n      * optional param of type `string` - prefix for the screenshot, enables multiple screenshots per test file\r\n      * optional param of type `number` - error threshold for image comparison, omit unless there is reason to increase it\r\n      * optional param of type `boolean` - enable strict antialiasing image comparison, screenshots on android have to be stretched, resulting in antialiased edges, which are ignored by default - enable for precise image comparison (e.g. pixelart)\r\n      * optional param of type `object` - crop the screenshot to specific region, defaults to whole 320x240 viewport\r\n   * `pointerMove(x,y)` - move mouse or touch input to specified location\r\n   * `pointerDown()` - press mouse or touch at previously moved-to location\r\n   * `pointerUp()` - depress mouse or touch at previously moved-to location\r\n   * `keyDown(key)` - trigger single keydown\r\n   * `keyUp(key)` - trigger single keyup\r\n\r\n## What do `waitBarrier` and `signalBarrier` do?\r\n\r\nThese two functions serve for synchronization points between the test page and your webdriver test.\r\nThe test page has to somehow signal when it is done rendering all relevant Crafty entities - that's what `signalBarrier(<label>)` is for.\r\nYou can then wait for a specific synchronization point with `waitBarrier(<label>)` in your webdriver test code - you can also increase the default wait timeout by passing an additional argument representing the timeout to wait in ms.\r\n\r\n## Miscellaneous\r\n\r\n* `supported-browsers.json` lists the browser we test on\r\n* This PR adds example test templates, not all tests covering every functionality, some tests will come in next PR. Additional tests can always be added over time.\r\n* fixes #950\r\n* save permanent test fixtures to tests/assets/webdriver and temporary test results to build/webdriver.\r\n* uploads screenshots of failed tests to Crafty-Distro/regression-tests.\r\n* streamlines Gruntfile.\r\n* changes all open sauce (in the cloud) tests to only work on pre-release `testing` branch.\r\n* In order to make it work for all browsers on Open Sauce, some dirty hacks have to be done:\r\n   * simulate synthetic events for those that do not support native ones\r\n   * rotate, crop and scale screenshots from android browsers\r\n\r\n\r\n## Example\r\n\r\n**template.html** - browser test page\r\n```html\r\n<!DOCTYPE html>\r\n<html>\r\n<head>\r\n  <meta charset=\"UTF-8\">\r\n  <title>CraftyJS Webdriver Test</title>\r\n  <link rel=\"stylesheet\" href=\"common.css\">\r\n  <script src='common.js'></script>\r\n  <script src='../../crafty.js'></script>\r\n</head>\r\n<body>\r\n<script>\r\nwindow.addEventListener('load', function() {\r\n  // ADD YOUR TEST CODE HERE; Crafty init'd with {w: 320, h: 240}\r\n\r\n  // green floor\r\n  Crafty.e(\"2D, DOM, Color, ground\")\r\n    .attr({ x: 0, y: 239, w: 320, h: 1 })\r\n    .color('rgb(0,255,0)');\r\n\r\n  // green platform\r\n  Crafty.e(\"2D, DOM, Color, ground\")\r\n    .attr({ x: 250, y: 120, w: 60, h: 1 })\r\n    .color('rgb(0,255,0)');\r\n\r\n  // blue player\r\n  var landedCount = 0;\r\n  Crafty.e(\"2D, DOM, Color, Gravity, Twoway, Button, Draggable\")\r\n    .attr({ x: 265, y: 140, w: 30, h: 30 })\r\n    .color('rgb(0,0,255)')\r\n    .gravityConst(1 * 50 * 50)\r\n    .twoway(5 * 50,17 * 50)\r\n    .gravity(\"ground\")\r\n    .bind(\"StartDrag\", function() {\r\n      this.antigravity();\r\n    })\r\n    .bind(\"StopDrag\", function() {\r\n      this.gravity();\r\n    })\r\n    .bind(\"LandedOnGround\", function(ground) {\r\n      landedCount++;\r\n      signalBarrier(\"landed\" + landedCount);\r\n    });\r\n\r\n}, false);\r\n</script>\r\n</body>\r\n</html>\r\n```\r\n\r\n**template.js** - node webdriver test\r\n```js\r\nQUnit.module(require('path').basename(module.filename, '.js'));\r\n\r\nQUnit.test(\"Multi test template part 1: Player lands on ground floor automatically after navigating to test page\", function(assert) {\r\n  return browser\r\n    .testUrl()\r\n    .waitBarrier(\"landed1\")\r\n    .assertResemble(\"firstLand\", 0.20);\r\n});\r\n\r\nQUnit.test(\"Multi test template part 2: Player lands on platform after jump key pressed\", function(assert) {\r\n  return browser\r\n    .keyDown('W').keyUp('W')\r\n    .waitBarrier(\"landed2\", 2000)\r\n    .assertResemble(\"secondLand\", 0.20);\r\n});\r\n\r\nQUnit.test(\"Multi test template part 3: Player lands on floor after being drag & dropped\", function(assert) {\r\n  return browser\r\n    .pointerMove(280, 107)\r\n    .pointerDown()\r\n    .pointerMove(25, 135)\r\n    .pointerUp()\r\n    .waitBarrier(\"landed3\", 2000)\r\n    .assertResemble(\"thirdLand\", 0.20);\r\n});\r\n\r\n```\r\n\r\n## Example screenshots\r\n\r\nProduces in step 1 this screenshot\r\n\r\n![template-multi-firstland-expected](https://cloud.githubusercontent.com/assets/3935691/11760025/4581da24-a08a-11e5-8158-9baca60f3665.png)\r\n\r\nProduces in step 2 this screenshot\r\n\r\n![template-multi-secondland-expected](https://cloud.githubusercontent.com/assets/3935691/11760029/4e18c968-a08a-11e5-8107-7cd9ef06f5e7.png)\r\n\r\nProduces in step 3 this screenshot\r\n\r\n![template-multi-thirdland-expected](https://cloud.githubusercontent.com/assets/3935691/11760031/575ce4f0-a08a-11e5-845b-933a094cf17d.png)\r\n\r\nAnd **checks** if the new screenshots match the previously recorded ones within margin of error.\r\n\r\nWhat do you think? This should give us much needed confidence when doing changes to graphics stuff.\r\n"},"comment":{"url":"https://api.github.com/repos/craftyjs/Crafty/issues/comments/164287930","html_url":"https://github.com/craftyjs/Crafty/pull/986#issuecomment-164287930","issue_url":"https://api.github.com/repos/craftyjs/Crafty/issues/986","id":164287930,"user":{"login":"starwed","id":34907,"avatar_url":"https://avatars.githubusercontent.com/u/34907?v=3","gravatar_id":"","url":"https://api.github.com/users/starwed","html_url":"https://github.com/starwed","followers_url":"https://api.github.com/users/starwed/followers","following_url":"https://api.github.com/users/starwed/following{/other_user}","gists_url":"https://api.github.com/users/starwed/gists{/gist_id}","starred_url":"https://api.github.com/users/starwed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/starwed/subscriptions","organizations_url":"https://api.github.com/users/starwed/orgs","repos_url":"https://api.github.com/users/starwed/repos","events_url":"https://api.github.com/users/starwed/events{/privacy}","received_events_url":"https://api.github.com/users/starwed/received_events","type":"User","site_admin":false},"created_at":"2015-12-13T19:00:49Z","updated_at":"2015-12-13T19:00:49Z","body":"Thanks for taking the time to not just implement this, but also thoroughly explain it!\r\n\r\nReading through your comments it sounds good to me.  Just to check my understanding: this particular PR implements the general framework as well as two specific DOM tests?\r\n\r\nOne theoretical possibility would be to compare DOM, Canvas, and WebGL rendering of the same scene against the same screenshot.  I have a [quasi-complicated scene](http://ludonomicon.com/crafty/triple-render.html) I've been using to check a variety of functionality in the three renderers simultaneously."}}