{"action":"created","issue":{"url":"https://api.github.com/repos/phetsims/joist/issues/210","labels_url":"https://api.github.com/repos/phetsims/joist/issues/210/labels{/name}","comments_url":"https://api.github.com/repos/phetsims/joist/issues/210/comments","events_url":"https://api.github.com/repos/phetsims/joist/issues/210/events","html_url":"https://github.com/phetsims/joist/issues/210","id":56122565,"number":210,"title":"Automatic test harness","user":{"login":"samreid","id":679486,"avatar_url":"https://avatars.githubusercontent.com/u/679486?v=3","gravatar_id":"","url":"https://api.github.com/users/samreid","html_url":"https://github.com/samreid","followers_url":"https://api.github.com/users/samreid/followers","following_url":"https://api.github.com/users/samreid/following{/other_user}","gists_url":"https://api.github.com/users/samreid/gists{/gist_id}","starred_url":"https://api.github.com/users/samreid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samreid/subscriptions","organizations_url":"https://api.github.com/users/samreid/orgs","repos_url":"https://api.github.com/users/samreid/repos","events_url":"https://api.github.com/users/samreid/events{/privacy}","received_events_url":"https://api.github.com/users/samreid/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/phetsims/joist/labels/high-priority","name":"high-priority","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"samreid","id":679486,"avatar_url":"https://avatars.githubusercontent.com/u/679486?v=3","gravatar_id":"","url":"https://api.github.com/users/samreid","html_url":"https://github.com/samreid","followers_url":"https://api.github.com/users/samreid/followers","following_url":"https://api.github.com/users/samreid/following{/other_user}","gists_url":"https://api.github.com/users/samreid/gists{/gist_id}","starred_url":"https://api.github.com/users/samreid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samreid/subscriptions","organizations_url":"https://api.github.com/users/samreid/orgs","repos_url":"https://api.github.com/users/samreid/repos","events_url":"https://api.github.com/users/samreid/events{/privacy}","received_events_url":"https://api.github.com/users/samreid/received_events","type":"User","site_admin":false},"milestone":null,"comments":27,"created_at":"2015-01-31T11:51:20Z","updated_at":"2015-07-02T18:40:22Z","closed_at":null,"body":"I'd like our team to have an automated test harness that launches each simulation, runs a mouse/touch fuzzer (or some other way of exercising some of the sim functionality) on it and summarizes the results.  It is getting very difficult to know whether common library changes will cause breaking changes in usage sites, and it is time consuming to manually test.  Even knowing if the simulation launches without crashing would, in many cases, be very helpful.\r\n\r\nPerhaps this can be done through the iframe API?  A parent frame that iterates through simulations: for each one, it launches the sim in an iframe and listens for messages that indicate the progress--if messages stall out, then it means the sim encountered a bug.  The sims should be run with ?ea\r\n\r\nThis should be designed so it could be run on any developer machine *or* a continuous integration machine (the latter may require PhantomJS?).\r\n\r\nA way to make this even sweeter would be to add temporary instrumentation into the changed library code so that the parent frame can collect statistics about how many times the modified code was actually hit in each simulation so we can get assurance that it has been tested.  Note: the tests described here are mainly to make sure things don't error out and crash--more manual tests would be required to confirm the behavior is correct.\r\n\r\nNot sure what repo such a thing should live in (if it should live at all)."},"comment":{"url":"https://api.github.com/repos/phetsims/joist/issues/comments/118120834","html_url":"https://github.com/phetsims/joist/issues/210#issuecomment-118120834","issue_url":"https://api.github.com/repos/phetsims/joist/issues/210","id":118120834,"user":{"login":"samreid","id":679486,"avatar_url":"https://avatars.githubusercontent.com/u/679486?v=3","gravatar_id":"","url":"https://api.github.com/users/samreid","html_url":"https://github.com/samreid","followers_url":"https://api.github.com/users/samreid/followers","following_url":"https://api.github.com/users/samreid/following{/other_user}","gists_url":"https://api.github.com/users/samreid/gists{/gist_id}","starred_url":"https://api.github.com/users/samreid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samreid/subscriptions","organizations_url":"https://api.github.com/users/samreid/orgs","repos_url":"https://api.github.com/users/samreid/repos","events_url":"https://api.github.com/users/samreid/events{/privacy}","received_events_url":"https://api.github.com/users/samreid/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T18:40:22Z","updated_at":"2015-07-02T18:40:22Z","body":"http://localhost:8080/chipper/tests/test-sims.html no longer works, did the URL change?"}}