{"action":"opened","issue":{"url":"https://api.github.com/repos/RIOT-OS/RIOT/issues/3363","labels_url":"https://api.github.com/repos/RIOT-OS/RIOT/issues/3363/labels{/name}","comments_url":"https://api.github.com/repos/RIOT-OS/RIOT/issues/3363/comments","events_url":"https://api.github.com/repos/RIOT-OS/RIOT/issues/3363/events","html_url":"https://github.com/RIOT-OS/RIOT/issues/3363","id":94330630,"number":3363,"title":"Proposal for directory structure for automated tests","user":{"login":"A-Paul","id":12623350,"avatar_url":"https://avatars.githubusercontent.com/u/12623350?v=3","gravatar_id":"","url":"https://api.github.com/users/A-Paul","html_url":"https://github.com/A-Paul","followers_url":"https://api.github.com/users/A-Paul/followers","following_url":"https://api.github.com/users/A-Paul/following{/other_user}","gists_url":"https://api.github.com/users/A-Paul/gists{/gist_id}","starred_url":"https://api.github.com/users/A-Paul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/A-Paul/subscriptions","organizations_url":"https://api.github.com/users/A-Paul/orgs","repos_url":"https://api.github.com/users/A-Paul/repos","events_url":"https://api.github.com/users/A-Paul/events{/privacy}","received_events_url":"https://api.github.com/users/A-Paul/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/RIOT-OS/RIOT/labels/CI-Infrastructure","name":"CI-Infrastructure","color":"fad8c7"},{"url":"https://api.github.com/repos/RIOT-OS/RIOT/labels/Request%20for%20Comments","name":"Request for Comments","color":"23F823"},{"url":"https://api.github.com/repos/RIOT-OS/RIOT/labels/tests","name":"tests","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-10T15:35:33Z","updated_at":"2015-07-10T15:35:33Z","closed_at":null,"body":"Define a directory under `$RIOTBASE/tests/` and outside `tests/unittests` where sources for automated tests can be placed. It has turned out in discussions [#3296](https://github.com/RIOT-OS/RIOT/pull/3296#issuecomment-118338031), [#3307](https://github.com/RIOT-OS/RIOT/pull/3307#discussion_r33860255) that the established *unittests* becomes insufficent for extended testing.\r\n\r\nMotivation\r\n----------\r\n\r\nThe agreement of a seperate directory structure should face the following difficulties.\r\n\r\n### tests/unittests ###\r\n\r\nThe tests in this directory are build together in one blob. This is implicating some weaknesses. \r\n\r\n+ With the increasing number of test cases the memory ressources of some boards are exhausted.\r\n+ It prevents the desired isolation of the running test cases.\r\n+ Necessary control of preconditions can not be archieved for test sequences that would require reinitializing (e.g `ng_ipv6_init()`)\r\n\r\n### tests ###\r\n\r\nThe directory is, beside of *unittests* home of several more or less freely named subdirectories for different test scenarios. It is not adviseable to mix the automated tests inbetween.\r\n\r\n+ Increases naming collisions or encourages creative naming to prevent them.\r\n+ Makes it difficult to keep track of locations.\r\n\r\nProposal\r\n--------\r\n\r\nThe idea is to name name subdirectory of tests(lets say *'automated'*) and put the tests in subdirectories. These may reflect the location of either the headers or the units source files to follow the \"principle of least suprise\".\r\nEvery *unit* or *test object* has further subdirs for sepeated test which require reflashing, etc...\r\n\r\n### Example for resulting structure ###\r\n(Reflecting loctions of C files)\r\n\r\n\r\n    tests/automated/core/atomic/\r\n    tests/automated/core/...\r\n    tests/automated/core/thread/\r\n    tests/automated/...\r\n    tests/automated/sys/auto_init/auto_init/\r\n    tests/automated/sys/...\r\n    tests/automated/sys/net/application_layer/ng_zep/ng_zep/\r\n    tests/automated/sys/net/...\r\n    tests/automated/sys/net/transport_layer/ng_udp/ng_udp/\r\n"}}