{"action":"opened","number":588,"pull_request":{"url":"https://api.github.com/repos/PrismJS/prism/pulls/588","id":36660409,"html_url":"https://github.com/PrismJS/prism/pull/588","diff_url":"https://github.com/PrismJS/prism/pull/588.diff","patch_url":"https://github.com/PrismJS/prism/pull/588.patch","issue_url":"https://api.github.com/repos/PrismJS/prism/issues/588","number":588,"state":"open","locked":false,"title":"[WIP] Implement test suite runner","user":{"login":"apfelbox","id":1032411,"avatar_url":"https://avatars.githubusercontent.com/u/1032411?v=3","gravatar_id":"","url":"https://api.github.com/users/apfelbox","html_url":"https://github.com/apfelbox","followers_url":"https://api.github.com/users/apfelbox/followers","following_url":"https://api.github.com/users/apfelbox/following{/other_user}","gists_url":"https://api.github.com/users/apfelbox/gists{/gist_id}","starred_url":"https://api.github.com/users/apfelbox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apfelbox/subscriptions","organizations_url":"https://api.github.com/users/apfelbox/orgs","repos_url":"https://api.github.com/users/apfelbox/repos","events_url":"https://api.github.com/users/apfelbox/events{/privacy}","received_events_url":"https://api.github.com/users/apfelbox/received_events","type":"User","site_admin":false},"body":"**This is a WIP, please don't merge it yet**\r\n\r\nI finally got around on implementing a simple (?) test suite runner. For now, the test suite runs in node, so just the language definitions will be tested, not the integration with the browser. We won't catch things like unsupported methods in old browser for now. (So, if your language definition uses `[].forEach` the test suite will pass (as node supports this), but IEold will fail as the method is unsupported there.)\r\n\r\n## Test case location\r\nAll tests are located in `/tests/languages/${language}/${file}`. `${language}` must match the name of the language definition in `components.js`. `${file}` is just a file name and only used to provide an info to the test runner.\r\n\r\nSo if the directory looks like this:\r\n```\r\n.\r\n└── languages\r\n    ├── css\r\n    │   └── blabla.css\r\n    └── javascript\r\n        └── testcase1.js\r\n```\r\n\r\nyour output will look like this:\r\n![screen shot 2015-06-01 at 20 51 10](https://cloud.githubusercontent.com/assets/1032411/7920531/03f74594-08a0-11e5-96dd-9b3241768987.png)\r\n\r\nAs you can see the file name doesn't matter, it is only included in the test description (that you can quickly find your failing tests).\r\n\r\n\r\n## Test case specification\r\n**Please provide feedback / better ideas than mine here.**\r\n\r\nAn example test case file looks like this:\r\n\r\n```\r\nvar a = 5;\r\n\r\n--------\r\n\r\n[\r\n\t[\"keyword\", \"var\"],\r\n\t[\"operator\", \"=\"],\r\n\t[\"number\", \"5\"],\r\n\t[\"punctuation\", \";\"]\r\n]\r\n```\r\n\r\nIt consists of\r\n\r\n    ... language snippet...\r\n    \r\n    /^----*\\w*$/m                        <-- yep, thats the exact regex that splits the test case file\r\n    \r\n    ... JSON of expected token stream ...\r\n\r\nPlease consider that the test case specification does use some simplifications / expectations. I tried to find a well enough mix of \"too much magic\" and \"way too much to write and manage\". Please be lenient with me here.\r\n\r\n1. The expected token stream is not an array of objects (like the result of `Prism.tokenize()`), but it is an array of two-element arrays. The only reason to do this is to reduce the amount of text you need to write for a test case.\r\n2. Only `token.type` and `token.content` are compared, something like `token.alias` is just ignored.\r\n3. The JSON part must be valid JSON. Although a little bit uncomfortable, this is my take against reinventing the wheel.\r\n4. If the test case file can't be properly parsed (the split produces more or less that two parts) the test case is failed.\r\n5. All non-matched parts (strings in the compiled token stream) are simply ignored.\r\n6. An entry in the expected token stream consists of `[${type}, ${content}]`\r\n7. The Prism test instances are isolated.\r\n\r\n\r\n## Implemention\r\nThe implemention looks like this:\r\n\r\n1. Load the complete test suite (= a nested array of all test case files).\r\n2. Define the test for all tests.\r\n3. Run the tests using mocha, chai and an isolated Prism instance.\r\n\r\nAs the language definitions (and Prism itself) doesn't use proper encapsulation (Prism is a global singleton), some hoop-jumping is required for this work: we need to run the source code in a vm in node (natively supported in `require(\"vm\")`, [docs](https://nodejs.org/api/vm.html)) and pass the context manually in their. This way we can build completely fresh and isolated Prism instances for each test (so that they won't interfere with each other). \r\n\r\nThis kills performance but as we can't just use `new Prism()`, this is required for ensuring that the tests run in isolation (even different tests in the same language should ideally use different instances – as they currently do).\r\n\r\nThis is not exactly ideal for performance... **but** let's write tests and worry about testing performance later! :tada:\r\n\r\n## ToDo\r\n\r\n[] test it / review it\r\n[] discuss the test case file format\r\n[] write tests for the languages (PRs welcome :sunglasses:)\r\n\r\n\r\nIntended to replace/refresh #355","created_at":"2015-06-01T19:12:04Z","updated_at":"2015-06-01T19:12:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/PrismJS/prism/pulls/588/commits","review_comments_url":"https://api.github.com/repos/PrismJS/prism/pulls/588/comments","review_comment_url":"https://api.github.com/repos/PrismJS/prism/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PrismJS/prism/issues/588/comments","statuses_url":"https://api.github.com/repos/PrismJS/prism/statuses/a806d1cc55f1d22d5a964c241bca40a74923763c","head":{"label":"apfelbox:mocha","ref":"mocha","sha":"a806d1cc55f1d22d5a964c241bca40a74923763c","user":{"login":"apfelbox","id":1032411,"avatar_url":"https://avatars.githubusercontent.com/u/1032411?v=3","gravatar_id":"","url":"https://api.github.com/users/apfelbox","html_url":"https://github.com/apfelbox","followers_url":"https://api.github.com/users/apfelbox/followers","following_url":"https://api.github.com/users/apfelbox/following{/other_user}","gists_url":"https://api.github.com/users/apfelbox/gists{/gist_id}","starred_url":"https://api.github.com/users/apfelbox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apfelbox/subscriptions","organizations_url":"https://api.github.com/users/apfelbox/orgs","repos_url":"https://api.github.com/users/apfelbox/repos","events_url":"https://api.github.com/users/apfelbox/events{/privacy}","received_events_url":"https://api.github.com/users/apfelbox/received_events","type":"User","site_admin":false},"repo":{"id":16993247,"name":"prism","full_name":"apfelbox/prism","owner":{"login":"apfelbox","id":1032411,"avatar_url":"https://avatars.githubusercontent.com/u/1032411?v=3","gravatar_id":"","url":"https://api.github.com/users/apfelbox","html_url":"https://github.com/apfelbox","followers_url":"https://api.github.com/users/apfelbox/followers","following_url":"https://api.github.com/users/apfelbox/following{/other_user}","gists_url":"https://api.github.com/users/apfelbox/gists{/gist_id}","starred_url":"https://api.github.com/users/apfelbox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apfelbox/subscriptions","organizations_url":"https://api.github.com/users/apfelbox/orgs","repos_url":"https://api.github.com/users/apfelbox/repos","events_url":"https://api.github.com/users/apfelbox/events{/privacy}","received_events_url":"https://api.github.com/users/apfelbox/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/apfelbox/prism","description":"Lightweight, robust, elegant syntax highlighting. A spin-off project from Dabblet. ","fork":true,"url":"https://api.github.com/repos/apfelbox/prism","forks_url":"https://api.github.com/repos/apfelbox/prism/forks","keys_url":"https://api.github.com/repos/apfelbox/prism/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apfelbox/prism/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apfelbox/prism/teams","hooks_url":"https://api.github.com/repos/apfelbox/prism/hooks","issue_events_url":"https://api.github.com/repos/apfelbox/prism/issues/events{/number}","events_url":"https://api.github.com/repos/apfelbox/prism/events","assignees_url":"https://api.github.com/repos/apfelbox/prism/assignees{/user}","branches_url":"https://api.github.com/repos/apfelbox/prism/branches{/branch}","tags_url":"https://api.github.com/repos/apfelbox/prism/tags","blobs_url":"https://api.github.com/repos/apfelbox/prism/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apfelbox/prism/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apfelbox/prism/git/refs{/sha}","trees_url":"https://api.github.com/repos/apfelbox/prism/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apfelbox/prism/statuses/{sha}","languages_url":"https://api.github.com/repos/apfelbox/prism/languages","stargazers_url":"https://api.github.com/repos/apfelbox/prism/stargazers","contributors_url":"https://api.github.com/repos/apfelbox/prism/contributors","subscribers_url":"https://api.github.com/repos/apfelbox/prism/subscribers","subscription_url":"https://api.github.com/repos/apfelbox/prism/subscription","commits_url":"https://api.github.com/repos/apfelbox/prism/commits{/sha}","git_commits_url":"https://api.github.com/repos/apfelbox/prism/git/commits{/sha}","comments_url":"https://api.github.com/repos/apfelbox/prism/comments{/number}","issue_comment_url":"https://api.github.com/repos/apfelbox/prism/issues/comments{/number}","contents_url":"https://api.github.com/repos/apfelbox/prism/contents/{+path}","compare_url":"https://api.github.com/repos/apfelbox/prism/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apfelbox/prism/merges","archive_url":"https://api.github.com/repos/apfelbox/prism/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apfelbox/prism/downloads","issues_url":"https://api.github.com/repos/apfelbox/prism/issues{/number}","pulls_url":"https://api.github.com/repos/apfelbox/prism/pulls{/number}","milestones_url":"https://api.github.com/repos/apfelbox/prism/milestones{/number}","notifications_url":"https://api.github.com/repos/apfelbox/prism/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apfelbox/prism/labels{/name}","releases_url":"https://api.github.com/repos/apfelbox/prism/releases{/id}","created_at":"2014-02-19T17:55:59Z","updated_at":"2015-02-21T23:20:12Z","pushed_at":"2015-06-01T18:39:35Z","git_url":"git://github.com/apfelbox/prism.git","ssh_url":"git@github.com:apfelbox/prism.git","clone_url":"https://github.com/apfelbox/prism.git","svn_url":"https://github.com/apfelbox/prism","homepage":"http://prismjs.com","size":1815,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"gh-pages"}},"base":{"label":"PrismJS:gh-pages","ref":"gh-pages","sha":"534f34cd35526481e4a32ac4d1eaf447a5a64bf2","user":{"login":"PrismJS","id":11140484,"avatar_url":"https://avatars.githubusercontent.com/u/11140484?v=3","gravatar_id":"","url":"https://api.github.com/users/PrismJS","html_url":"https://github.com/PrismJS","followers_url":"https://api.github.com/users/PrismJS/followers","following_url":"https://api.github.com/users/PrismJS/following{/other_user}","gists_url":"https://api.github.com/users/PrismJS/gists{/gist_id}","starred_url":"https://api.github.com/users/PrismJS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrismJS/subscriptions","organizations_url":"https://api.github.com/users/PrismJS/orgs","repos_url":"https://api.github.com/users/PrismJS/repos","events_url":"https://api.github.com/users/PrismJS/events{/privacy}","received_events_url":"https://api.github.com/users/PrismJS/received_events","type":"Organization","site_admin":false},"repo":{"id":4979467,"name":"prism","full_name":"PrismJS/prism","owner":{"login":"PrismJS","id":11140484,"avatar_url":"https://avatars.githubusercontent.com/u/11140484?v=3","gravatar_id":"","url":"https://api.github.com/users/PrismJS","html_url":"https://github.com/PrismJS","followers_url":"https://api.github.com/users/PrismJS/followers","following_url":"https://api.github.com/users/PrismJS/following{/other_user}","gists_url":"https://api.github.com/users/PrismJS/gists{/gist_id}","starred_url":"https://api.github.com/users/PrismJS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrismJS/subscriptions","organizations_url":"https://api.github.com/users/PrismJS/orgs","repos_url":"https://api.github.com/users/PrismJS/repos","events_url":"https://api.github.com/users/PrismJS/events{/privacy}","received_events_url":"https://api.github.com/users/PrismJS/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PrismJS/prism","description":"Lightweight, robust, elegant syntax highlighting.","fork":false,"url":"https://api.github.com/repos/PrismJS/prism","forks_url":"https://api.github.com/repos/PrismJS/prism/forks","keys_url":"https://api.github.com/repos/PrismJS/prism/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PrismJS/prism/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PrismJS/prism/teams","hooks_url":"https://api.github.com/repos/PrismJS/prism/hooks","issue_events_url":"https://api.github.com/repos/PrismJS/prism/issues/events{/number}","events_url":"https://api.github.com/repos/PrismJS/prism/events","assignees_url":"https://api.github.com/repos/PrismJS/prism/assignees{/user}","branches_url":"https://api.github.com/repos/PrismJS/prism/branches{/branch}","tags_url":"https://api.github.com/repos/PrismJS/prism/tags","blobs_url":"https://api.github.com/repos/PrismJS/prism/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PrismJS/prism/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PrismJS/prism/git/refs{/sha}","trees_url":"https://api.github.com/repos/PrismJS/prism/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PrismJS/prism/statuses/{sha}","languages_url":"https://api.github.com/repos/PrismJS/prism/languages","stargazers_url":"https://api.github.com/repos/PrismJS/prism/stargazers","contributors_url":"https://api.github.com/repos/PrismJS/prism/contributors","subscribers_url":"https://api.github.com/repos/PrismJS/prism/subscribers","subscription_url":"https://api.github.com/repos/PrismJS/prism/subscription","commits_url":"https://api.github.com/repos/PrismJS/prism/commits{/sha}","git_commits_url":"https://api.github.com/repos/PrismJS/prism/git/commits{/sha}","comments_url":"https://api.github.com/repos/PrismJS/prism/comments{/number}","issue_comment_url":"https://api.github.com/repos/PrismJS/prism/issues/comments{/number}","contents_url":"https://api.github.com/repos/PrismJS/prism/contents/{+path}","compare_url":"https://api.github.com/repos/PrismJS/prism/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PrismJS/prism/merges","archive_url":"https://api.github.com/repos/PrismJS/prism/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PrismJS/prism/downloads","issues_url":"https://api.github.com/repos/PrismJS/prism/issues{/number}","pulls_url":"https://api.github.com/repos/PrismJS/prism/pulls{/number}","milestones_url":"https://api.github.com/repos/PrismJS/prism/milestones{/number}","notifications_url":"https://api.github.com/repos/PrismJS/prism/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PrismJS/prism/labels{/name}","releases_url":"https://api.github.com/repos/PrismJS/prism/releases{/id}","created_at":"2012-07-10T19:47:14Z","updated_at":"2015-05-31T05:16:08Z","pushed_at":"2015-06-01T16:49:43Z","git_url":"git://github.com/PrismJS/prism.git","ssh_url":"git@github.com:PrismJS/prism.git","clone_url":"https://github.com/PrismJS/prism.git","svn_url":"https://github.com/PrismJS/prism","homepage":"http://prismjs.com","size":6358,"stargazers_count":2199,"watchers_count":2199,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":313,"mirror_url":null,"open_issues_count":130,"forks":313,"open_issues":130,"watchers":2199,"default_branch":"gh-pages"}},"_links":{"self":{"href":"https://api.github.com/repos/PrismJS/prism/pulls/588"},"html":{"href":"https://github.com/PrismJS/prism/pull/588"},"issue":{"href":"https://api.github.com/repos/PrismJS/prism/issues/588"},"comments":{"href":"https://api.github.com/repos/PrismJS/prism/issues/588/comments"},"review_comments":{"href":"https://api.github.com/repos/PrismJS/prism/pulls/588/comments"},"review_comment":{"href":"https://api.github.com/repos/PrismJS/prism/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PrismJS/prism/pulls/588/commits"},"statuses":{"href":"https://api.github.com/repos/PrismJS/prism/statuses/a806d1cc55f1d22d5a964c241bca40a74923763c"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":3,"additions":292,"deletions":2,"changed_files":7}}