There’s an interesting design constraint I encountered fairly early on when writing a purely C-based unit testing framework that I hadn’t really thought much about when using testing frameworks in other languages like C++, C#, and Java. How do you design a testing framework where resources are cleaned up regardless of a test passing or failing? Let me give an example using googletest.
TEST(ResourceCleanupExamples, CleanupOnTestFailure) {
auto resource = std::make_unique<int>(5);
ASSERT_EQ(*resource, 10);
}The resource variable’s memory is released even though the ASSERT_EQ statement fails. This occurs because of the RAII nature of modern C++. Other languages, of course, have garbage collection and other mechanisms to clean up test resources. In a purely C-based environment everything is much more explicit. Every resource acquisition typically also has an associated line where that resource is explicitly released. Automatic cleanup of all resources on scope exit is not always possible (even though you can have some cleanup on scope exit). The same example could be rewritten in otter’s test framework to demonstrate the problem.
OTTER_TEST(cleanup_on_test_failure) {
int *resource = malloc(sizeof(*resource));
OTTER_ASSERT(resource != NULL);
*resource = 5;
OTTER_ASSERT(*resource == 10);
free(resource); /* This line will never run */
}A unit test in otter needs an additional piece because of this problem. It’s the OTTER_TEST_END macro as the very last line. Any cleanup logic within is guaranteed to execute regardless of the test passing or failing (i.e., prematurely exiting the test).
OTTER_TEST(cleanup_on_test_failure) {
int *resource = malloc(sizeof(*resource));
OTTER_ASSERT(resource != NULL);
*resource = 5;
OTTER_ASSERT(*resource == 10);
OTTER_TEST_END(
/* Guaranteed to run even when assertions fail */
free(resource);
);
}This expands to something like this. The OTTER_ASSERT macro expands to blocks that set information about the particular assertion failure in the otter_test_context struct. It then uses a goto to jump to the end of the test where all the cleanup occurs. That way, the label, otter_test_end will be executed no matter what assertion fails. We will also fall into the label if all assertions pass. Any resource cleanup specified will occur.
bool cleanup_on_test_failure(otter_test_context *otter_test_ctx) {
int *resource = malloc(sizeof(*resource));
if (!(resource != NULL)) {
(otter_test_ctx)->test_status = false;
(otter_test_ctx)->failed_expression = "resource != NULL";
(otter_test_ctx)->failed_file = "src/array_tests.c";
(otter_test_ctx)->failed_line = 26;
goto otter_test_end;
};
*resource = 5;
if (!(*resource == 10)) {
(otter_test_ctx)->test_status = false;
(otter_test_ctx)->failed_expression = "*resource == 10";
(otter_test_ctx)->failed_file = "src/array_tests.c";
(otter_test_ctx)->failed_line = 29;
goto otter_test_end;
};
otter_test_end:
do {
free(resource);
} while (0);
return (otter_test_ctx)->test_status;
}
There are other designs I considered. What if OTTER_ASSERT could be checked in the unit test and cleanup logic could be put inside the true block? What about introducing fixtures that are generated for each test invocation and know how to clean themselves up? For shared setup logic, fixtures are still something I want to explore implementing. I want each feature I add to be simple to use. It should be non-verbose from the usage side of things.
Leave a Reply