# Creating atf-based tests for NetBSD src This quick tutorial is an attempt to workaround the lack of proper documentation in atf. The tutorial provides a guideline on how to start creating new test programs and/or test cases, how these tests are tied to the NetBSD source tree and a short reference of the most commonly used functions. You should start by reading the [tests(7)](http://netbsd.gw.com/cgi-bin/man-cgi?tests++NetBSD-current) manual page, which is probably the only sane document in the whole documentation. Any other attempts at reading the atf-* manual pages are probably doomed unless you are already familiar with atf itself and its internals. Still, you may be able to get some useful information out of [atf-run(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-run++NetBSD-current), [atf-report(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-report++NetBSD-current), [atf-test-program(1)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-test-program++NetBSD-current), [atf-c-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-c-api++NetBSD-current) and [atf-sh-api(3)](http://netbsd.gw.com/cgi-bin/man-cgi?atf-sh-api++NetBSD-current). **IMPORTANT: Do not take anything for granted, SPECIALLY if you have previously worked with and/or have seen src/regress/. Your assumptions are most likely wrong.** ## Test programs vs. test cases So, what is what and how do you organize your tests? A **test case** is a piece of code that exercises a particular functionality of another piece of code. Commonly, test cases validate the outcome of a particular source function or class method, the validity of the execution of a command with a particular combination of flags/arguments, etc. Test cases are supposed to be very concise, in the sense that they should just be testing *one behavior*. A **test program** is a binary that collects and exposes a group of test cases. Typically, these test programs expose conceptually-related tests or all the tests for a particular source file. In general, having many test programs with **just one test case** in them is **wrong** and smells from the previous layout of src/regress/. Think about some other organization. And don't blame atf for this separation: this is extremely common in (almost?) all other test frameworks and, when used wisely, becomes an invaluable classification. For example, suppose you have the following fictitious source files for the ls tool: * bin/ls/fs.c: Provides the list_files() and stat_files() functions. * bin/ls/ui.c: Provides the format_columns() function. * bin/ls/main.c: The main method for ls. Then, you could define the following test programs and test cases: * bin/ls/fs_test.c: Provides test cases for list_files and stat_files. These would be named list_files__empty_directory, list_files__one_file, list_files__multiple_files, stat_files__directory, stat_files__symlink, etc. * bin/ls/ui_test.c: Provides test cases for the format_columns function. These would be named format_columns__no_files, format_columns__multiple_files, etc. * bin/ls/integration_test.sh: Provides "black box" test cases for the binary itself. These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc. Try to keep your test case names as descriptive as possible. ## Adding a new test To add a new *test case* to the source tree, look for any test program in src/tests/ that can assimilate it. If you find such a program, just add the test case to it; no other changes are required. Otherwise, you will have to create a new test program. To add a new *test program* to the source tree: 1. Locate the appropriate subdirectory in which to put your test program. It is OK (and **expected**) to have multiple test programs into the same directory. **Restrain yourself from creating one directory per test program.** If the subdirectory exists: 1. Choose a sane name for the test program; the name must not be so specific that it restricts the addition of future test cases into it. 1. Create the test program source file using one of the templates below. E.g. src/tests/tutorial/sample_test.c. 1. Add the new test program to the Makefile. If the subdirectory does not exist: 1. Do the same as above. 1. Create the Makefile for the directory using the templates below. 1. Edit the parent Makefile to recurse into the new subdirectory. 1. Edit src/etc/mtree/NetBSD.base.dist to register the new subdirectory. Your test will be installed under /usr/tests/. 1. Edit src/distrib/sets/lists/tests/mi to register the new test program. Do not forget to add .debug entries if your test program is a C/C++ binary. ### Makefile template # $NetBSD: atf.mdwn,v 1.1 2010/09/01 20:59:24 jmmv Exp $ .include # This must always be defined. TESTSDIR= ${TESTSBASE}/bin/ls # Define only the variables you actually need for the directory. TESTS_C+= c1_test c2_test # Correspond to c1_test.c and c2_test.c. TESTS_SH+= sh1_test sh2_test # Correspond to sh1_test.c and sh2_test.c # Define only if your tests need any data files. FILESDIR= ${TESTSDIR} FILES= testdata1.txt testdata2.bin # Any necessary data files. .include ## C test programs ### Template #include ATF_TC(tc, my_test_case); ATF_TC_HEAD(tc, my_test_case) { atf_tc_set_md_var(tc, "descr", "This test case ensures that..."); } ATF_TC_BODY(tc, my_test_case) { ATF_CHECK(returns_a_boolean()); /* Non-fatal test. */ ATF_REQUIRE(returns_a_boolean()); /* Non-fatal test. */ ATF_CHECK_EQ(4, 2 + 2); /* Non-fatal test. */ ATF_REQUIRE_EQ(4, 2 + 2); /* Fatal test. */ if (!condition) atf_tc_fail("Condition not met!"); /* Explicit failure. */ } ATF_TP_ADD_TCS(tp) { ATF_TP_ADD_TC(tp, my_test_case); } ### How to build To build a C test program, append the name of the test program (without the .c extension) to the TESTS_C variable in the Makefile. For example: .include TESTSDIR= ${TESTSBASE}/bin/ls TESTS_C+= fs_test ui_test .include ## Shell test programs ### Template atf_test_case my_test_case my_test_case_head() { atf_set "descr" "This test case ensures that..." } my_test_case_body() { touch file1 file2 cat >expout < TESTSDIR= ${TESTSBASE}/bin/ls TESTS_SH+= integration_test something_else_test .include If you want to run the test program yourself, you should know that shell-based test programs are processed with the atf-sh interpreter. atf-sh is just a thin wrapper over /bin/sh that loads the shared atf code and then delegates execution to your source file. ## FAQ ### How do I atfify a plain test program? Let's suppose you have a program to exercise a particular piece of code. Conceptually this implements a test but it does not use atf at all. For example: #include #include #include #include /* This test program exercises the snprintf function. */ int main(void) { char buf[1024]; printf("Testing integers"); snprintf(buf, sizeof(buf), "%d", 3); if (strcmp(buf, "3") != 0) errx(EXIT_FAILURE, "%d failed"); snprintf(buf, sizeof(buf), "a %d b", 5); if (strcmp(buf, "a 5 b") != 0) errx(EXIT_FAILURE, "%d failed"); printf("Testing strings"); snprintf(buf, sizeof(buf), "%s", "foo"); if (strcmp(buf, "foo") != 0) errx(EXIT_FAILURE, "%s failed"); snprintf(buf, sizeof(buf), "a %s b", "bar"); if (strcmp(buf, "a bar b") != 0) errx(EXIT_FAILURE, "%s failed"); return EXIT_SUCCESS; } To convert this program into an atf test program, use the template above and keep this in mind: * Split the whole main function into separate test cases. In this scenario, the calls to printf(3) delimit a good granularity for the test cases: one for the integer formatter, one for the string formatter, etc. * Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and report errors. errx should not be used. The result would look like: #include #include ATF_TC(tc, integer_formatter); ATF_TC_HEAD(tc, integer_formatter) { atf_tc_set_md_var(tc, "descr", "Validates the %d formatter"); } ATF_TC_BODY(tc, integer_formatter) { char buf[1024]; snprintf(buf, sizeof(1024), "%d", 3); ATF_CHECK_STREQ("3", buf); snprintf(buf, sizeof(1024), "a %d b", 5); ATF_CHECK_STREQ("a 5 b", buf); } ATF_TC(tc, string_formatter); ATF_TC_HEAD(tc, string_formatter) { atf_tc_set_md_var(tc, "descr", "Validates the %s formatter"); } ATF_TC_BODY(tc, string_formatter) { char buf[1024]; snprintf(buf, sizeof(1024), "%s", "foo"); ATF_CHECK_STREQ("foo", buf); snprintf(buf, sizeof(1024), "a %s b", "bar"); ATF_CHECK_STREQ("a bar b", buf); } ATF_TP_ADD_TCS(tp) { ATF_TP_ADD_TC(tp, integer_formatter); ATF_TP_ADD_TC(tp, string_formatter); } Which can later be invoked as any of: $ ./snprintf_test integer_formatter $ ./snprintf_test string_formatter $ atf-run snprintf_test | atf-report ### Do I need to remove temporary files? No. atf-run does this automatically for you, because it runs every test program in its own temporary subdirectory.