{"action":"created","issue":{"url":"https://api.github.com/repos/KWARC/MMT/issues/101","labels_url":"https://api.github.com/repos/KWARC/MMT/issues/101/labels{/name}","comments_url":"https://api.github.com/repos/KWARC/MMT/issues/101/comments","events_url":"https://api.github.com/repos/KWARC/MMT/issues/101/events","html_url":"https://github.com/KWARC/MMT/issues/101","id":118772229,"number":101,"title":"archive-level regression test functionality","user":{"login":"kohlhase","id":4037538,"avatar_url":"https://avatars.githubusercontent.com/u/4037538?v=3","gravatar_id":"","url":"https://api.github.com/users/kohlhase","html_url":"https://github.com/kohlhase","followers_url":"https://api.github.com/users/kohlhase/followers","following_url":"https://api.github.com/users/kohlhase/following{/other_user}","gists_url":"https://api.github.com/users/kohlhase/gists{/gist_id}","starred_url":"https://api.github.com/users/kohlhase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kohlhase/subscriptions","organizations_url":"https://api.github.com/users/kohlhase/orgs","repos_url":"https://api.github.com/users/kohlhase/repos","events_url":"https://api.github.com/users/kohlhase/events{/privacy}","received_events_url":"https://api.github.com/users/kohlhase/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/KWARC/MMT/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/KWARC/MMT/labels/help%20wanted","name":"help wanted","color":"159818"}],"state":"open","locked":false,"assignee":{"login":"cmaeder","id":6043580,"avatar_url":"https://avatars.githubusercontent.com/u/6043580?v=3","gravatar_id":"","url":"https://api.github.com/users/cmaeder","html_url":"https://github.com/cmaeder","followers_url":"https://api.github.com/users/cmaeder/followers","following_url":"https://api.github.com/users/cmaeder/following{/other_user}","gists_url":"https://api.github.com/users/cmaeder/gists{/gist_id}","starred_url":"https://api.github.com/users/cmaeder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmaeder/subscriptions","organizations_url":"https://api.github.com/users/cmaeder/orgs","repos_url":"https://api.github.com/users/cmaeder/repos","events_url":"https://api.github.com/users/cmaeder/events{/privacy}","received_events_url":"https://api.github.com/users/cmaeder/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-11-25T06:37:56Z","updated_at":"2015-11-25T07:52:07Z","closed_at":null,"body":"In https://github.com/KWARC/sTeX/issues/193 I have sketched the first ideas for regression testing in sTeX. I think I need MMT support for this. Maybe \"need\" is the wrong word, but regression testing support would be useful for other \"regression-testing by repository\" situations, like MMT.  \r\n\r\nHere is the workflow I envision (using sTeX as an example and the ```HelloWorld/smglom``` repository). \r\n0. we make a new archive dimension ```test```, into that we copy all the generated OMDoc files in ```HelloWorld/smglom```  (preserving the directory structure) that we want to participate in the test. Alternatively, we can also write these OMDoc files by hand, if we cannot generate them yet. \r\n1. on every commit to the archive, ```travis``` builds ```HelloWorld/smglom``` and reports any build errors. \r\n 2. travis compares the contents of the ```test``` dimension to the build results and reports any discrepancies (we may need clever comparison tools here eventually)\r\n3. If there are no discrepancies, then all is fine and travis exits successfully. \r\n4. If there are discrepancies the committer and the sTeX maintainer get notified and look at the differences.\r\n5. If the discrepancies are legitimate/intended (e.g. when the sTeX bindings are improved and therfore yield better results), then we copy them to the ```test``` dimension (they become the new gold standard) and commit (thus restarting at 1.).  \r\n6. if discrepancies are not legitimate/intendednot, we fix the sTeX bindings, commit and start at 1. again.  \r\nI think that MMT should support this workflow to make regression testing easier and more common. "},"comment":{"url":"https://api.github.com/repos/KWARC/MMT/issues/comments/159527569","html_url":"https://github.com/KWARC/MMT/issues/101#issuecomment-159527569","issue_url":"https://api.github.com/repos/KWARC/MMT/issues/101","id":159527569,"user":{"login":"kohlhase","id":4037538,"avatar_url":"https://avatars.githubusercontent.com/u/4037538?v=3","gravatar_id":"","url":"https://api.github.com/users/kohlhase","html_url":"https://github.com/kohlhase","followers_url":"https://api.github.com/users/kohlhase/followers","following_url":"https://api.github.com/users/kohlhase/following{/other_user}","gists_url":"https://api.github.com/users/kohlhase/gists{/gist_id}","starred_url":"https://api.github.com/users/kohlhase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kohlhase/subscriptions","organizations_url":"https://api.github.com/users/kohlhase/orgs","repos_url":"https://api.github.com/users/kohlhase/repos","events_url":"https://api.github.com/users/kohlhase/events{/privacy}","received_events_url":"https://api.github.com/users/kohlhase/received_events","type":"User","site_admin":false},"created_at":"2015-11-25T07:52:07Z","updated_at":"2015-11-25T07:52:07Z","body":"I have added the ```test``` dimension for ```HelloWorld/smglom``` (where all tests should be OK), so we have start developing here the MMT/travis functionality here. "}}