How to Minimize Test Cases for Bugs

In order for a GCC developer to fix a bug, the bug must be reproducible by means of a self-contained test case. Our bug reporting instructions ask for the preprocessed version of the file that triggers the bug. Often this file is very large; there are several reasons for making it as small as possible:

• Removal of proprietary sources may be required to allow publication.
• Test cases are more easily reproducible if there are fewer dependencies on target system features, such as system headers.
• Smaller test cases make debugging easier.
• GCC developers prefer bug reports with small, portable test cases.
• Minimized test cases can be added to the GCC testsuite.

There are basically two methods: either directly construct a new testcase that reproduces the bug, or iteratively reduce the original testcase. It is possible to automatically reduce testcases. Most test cases can be reduced to fewer than 30 lines!

Direct approach

Always try a direct approach before resorting to the brute-force method of minimizing a large test case:

• Write a small test case based on the code at the location reported in an error message. If the test needs to call a library function (e.g., printf), specify the function's declaration directly rather than including a header file.
• For suspected bugs in a GCC runtime library, write a small test case that calls that function, either directly or indirectly, with the arguments that cause it to fail. It's OK to include header files that are part of the GCC release.

Brute force approach

The brute force approach is simply an iterative method to cut down on the size of the test case by deleting chunks of code. After each attempt to shorten the test case, check whether the bug still is evident. If not, back out to the previous version that demonstrated the bug. For this, either use the undo function of your editor, or use frequent backup files; alternatively, you can use #if 0 and #endif. The automatic and recommended way to reduce a testcase is using the C-Vise or the C-Reduce tool.

If you have access to the original sources, it is better to start with the original sources and when those cannot be reduced further, generate and reduce the preprocessed sources. Both methods are described below.

Stripping the original code

Copy the original source file, plus header files that you might need to modify, to a separate location where you can duplicate the failure. Try the following, backing up from the unsuccessful attempts.

• Delete all functions that follow the one that triggers the failure.
  • If it's in an inline function, remove the inline directive or write a dummy function to call it.
  • If it's a template function, provide an explicit instantiation.
• Strip as much code as possible from the function that triggers the failure; the remaining code doesn't need to make sense.
  • Remove function calls.
  • Remove C++ I/O. If you need output to demonstrate the bug, use printf and declare it directly: extern "C" int printf(const char *, ...);.
  • Remove unused variables.
  • Replace user-defined types (structs, classes) with built-in types (int, double).
  • Replace typedefs with primitive types.
• Delete functions that come before the function that triggers the failure, or replace their definitions with declarations.
• Move includes of system headers before other headers.
• Remove #include directives, or replace them with only the code that is needed from the included file to compile the main file.
• Replace C++ templates with regular functions and classes, or get rid of as many template arguments as possible.
• Simplify class hierarchies: Remove base classes, or use base classes instead of derived ones.
• Iterate through the following:
  • Remove unused class and struct members.
  • Remove definitions of unused data types.
  • Remove includes of unneeded header files.

Usually, the file will now be down to fewer than 30 lines, plus includes. Repeat the same steps with remaining included header files. To reduce the number of files you are working on, you can directly include the content of the header file into the source file you are working on.

In the next step, run the compiler with -save-temps to get the preprocessed source file.

Stripping preprocessed sources

The preprocessed file contains lines starting with # that tell the compiler the original file and line number of each line of code. Remove these file and line directives from the preprocessed file so that the compiler will report locations in the preprocessed file, using one of the following:

• cat bug.ii | grep -v '^# .*$' | grep -v '^[[:space:]]*$' > bug2.ii
• perl -pi -e 's/^#.*\n//g;' bug.ii
• sed '/^#/d' bug.ii > bug2.ii
• within vi: :g/^#/d

The preprocessed sources will now consist largely of header files. Follow the same steps as for stripping the original code, with these additional techniques.

• There will be large stretches of typedefs from system headers and, for C++, things enclosed in namespaces. Delete large chunks working from the bottom up, e.g. whole extern "C" blocks, or blocks enclosed in namespace XXX {...}.
• If you get a new compiler error due to a missing definition for a function or type that you deleted, replace the whole deleted chunk with only the missing declaration, or modify the code to no longer use it.

At this stage, you will be able to delete chunks of hundreds or even thousands of lines at a time and can quickly reduce the preprocessed sources to something small.