{"action":"created","issue":{"url":"https://api.github.com/repos/gap-system/gap/issues/314","labels_url":"https://api.github.com/repos/gap-system/gap/issues/314/labels{/name}","comments_url":"https://api.github.com/repos/gap-system/gap/issues/314/comments","events_url":"https://api.github.com/repos/gap-system/gap/issues/314/events","html_url":"https://github.com/gap-system/gap/issues/314","id":113773080,"number":314,"title":"Improve coverage of the test suite","user":{"login":"fingolfin","id":241512,"avatar_url":"https://avatars.githubusercontent.com/u/241512?v=3","gravatar_id":"","url":"https://api.github.com/users/fingolfin","html_url":"https://github.com/fingolfin","followers_url":"https://api.github.com/users/fingolfin/followers","following_url":"https://api.github.com/users/fingolfin/following{/other_user}","gists_url":"https://api.github.com/users/fingolfin/gists{/gist_id}","starred_url":"https://api.github.com/users/fingolfin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fingolfin/subscriptions","organizations_url":"https://api.github.com/users/fingolfin/orgs","repos_url":"https://api.github.com/users/fingolfin/repos","events_url":"https://api.github.com/users/fingolfin/events{/privacy}","received_events_url":"https://api.github.com/users/fingolfin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/gap-system/gap/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-10-28T09:05:01Z","updated_at":"2015-10-31T18:13:13Z","closed_at":null,"body":"I think one of the major problems in GAP development is that we lack a comprehensive test suite. Hence, any given change, no matter how innocent it may appear, can break functionality in GAP, and it may take us months until we notice it, say after a user reports an issue.\r\n\r\nLet's try to fix that. We discussed this often enough, so, I'll try to summarize what I think the conclusion of those discussions is. This issue here then could be considered a \"meta-issue\", tracking the overal progress, with the actual work done elsewhere.\r\n\r\n1. The test setup should be improved, to make running tests easier. See issue #260.\r\n\r\n2. Of course, *adding* new tests should also be easy. I believe that our current test system is very useful, but is also limited in some regards, so perhaps we need to add some new functionality there (not to replace the existing test system, but to augment it). My `UnitTests` package was an experiment in that direction, and I feel there are some other ideas we could explore in this regard. But that should not be discussed here, but rather on a separate issue and/or the mailing lists, and/or in live meetings.\r\n\r\n3. We need a way to measure test coverage. @ChrisJefferson wrote code that can generate webpage that shows for each file in the GAP library whether it is hit by any test or not. Of course line coverage is not as good as actually thinking in detail about each function and whether every situation in it is properly covered; but it is still tremendously better than not having any way to find out which code is tested or not. Therefore, I think we should make heavy use of this feature. In particular:\r\n\r\n   1. Make sure it is as easy as possible to use, and well-documented, so that devs can and will uses it on their machines regularly (not just for testing GAP's test coverage, but also of their packages)\r\n\r\n   2. Have a sever somewhere which automatically runs this regulalry, and provides the results easily accessible to everybody. Ideally, we'd also track some aggregate coverage numbers over time. This way, people can watch progress on the code coverage. This can be a huge motivation (think \"gameification\").\r\n\r\n4. Get people to actually write *meaningful* tests. I know that the previous three points would already help motivate me to invest some time into that. But in the end, it's a huge project. But it might be something that could be used to get people involved in the project. Of course there are some functions were you really need to know what's going on in order to properly (and *correctly*) test them, but right now, I believe there are tons of low-hanging fruits. E.g. writing tests for the ZmodnZ code in all its variants.\r\nPerhaps some people can also get fund for this, and hire some students to help out with this. Let's be creative!"},"comment":{"url":"https://api.github.com/repos/gap-system/gap/issues/comments/152757400","html_url":"https://github.com/gap-system/gap/issues/314#issuecomment-152757400","issue_url":"https://api.github.com/repos/gap-system/gap/issues/314","id":152757400,"user":{"login":"ChrisJefferson","id":811527,"avatar_url":"https://avatars.githubusercontent.com/u/811527?v=3","gravatar_id":"","url":"https://api.github.com/users/ChrisJefferson","html_url":"https://github.com/ChrisJefferson","followers_url":"https://api.github.com/users/ChrisJefferson/followers","following_url":"https://api.github.com/users/ChrisJefferson/following{/other_user}","gists_url":"https://api.github.com/users/ChrisJefferson/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisJefferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisJefferson/subscriptions","organizations_url":"https://api.github.com/users/ChrisJefferson/orgs","repos_url":"https://api.github.com/users/ChrisJefferson/repos","events_url":"https://api.github.com/users/ChrisJefferson/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisJefferson/received_events","type":"User","site_admin":false},"created_at":"2015-10-31T18:13:13Z","updated_at":"2015-10-31T18:13:13Z","body":"Some documentation would be good, more is coming :)\r\n\r\nGreen = line read and executed\r\nRed = line read, not executed\r\nGrey = line not read or executed (lines with code on that are 'not read', don't count as stored code to GAP's internals. I'm going to explain this better in some documentaiton).\r\n\r\nIn short -- we want no red on either profile."}}