CppCon 2020

Although various frameworks support the writing of unit tests, creating a well designed effective unit test suite is still a non-trivial manual work. In this presentation we show how fuzzing can be used to automatically generate and run test cases based only on the class' interface.

Imagine you want test a container like the std::deque, essentially a vector of fix sized buffers with push/pop operations at both ends. You write tests for the empty container, then for push_back, but then how to continue? Write test for push_back then pop_back or push_back then pop_front and then push_front etc... We see that there is an explosion of the number of possible test cases with regarding all the interface operations. Should we write all the combinations of all interface calls? Is there a value writing a test with 5 or 15 push_backs? Will we miss to write test for 16 push_backs when the implementation is forced to allocate new block?

In this presentation we show a method when the clang libFuzzer library is used to generate and run unit test cases automatically based on the class interface description. Instead of using the libFuzzer generated byte string as an input, we interpret it as a list of member function calls and their arguments. Driven by the coverage guidance of libFuzzer, we are able to generate more and more complex combinations of interface calls. Giving optional preconditions for the interface functions we can filter out invalid combinations, like calling pop_back before inserting elements to a container. We check class invariants between every method call and various sanitizers can also be applied. The implementation uses only language native solutions, no code generations or external scripts are required.

Like other fuzzer solutions this automatic tester can run for arbitrary time to discover edge cases the programmer ignored. The close to minimal subset of discovered test cases providing the maximum coverage can be stored as a "regression test" to re-run later. The implemented prototype performed well both on artificial test cases and real world C++ containers. The prototype is available at: https://gitlab.com/wilzegers/autotest