Practical Testing of a C99 Compiler Using Output Comparison
This provides a practical method for compiler developers to improve testing efficiency and bug detection, though it appears incremental rather than paradigm-shifting.
The authors tackled compiler testing by comparing outputs with existing tools, enabling bulk test case addition from existing sources. This approach found hundreds of bugs in various compilers, including 18 in GNU C compiler and critical issues in ARM versions.
A simple technique is presented for testing a C99 compiler, by comparison of its output with output from preexisting tools. The advantage to this approach is that new test cases can be added in bulk from existing sources, reducing the need for in-depth investigation of correctness issues, and for creating new test code by hand. This technique was used in testing the PalmSource Palm OS Cobalt ARM C/C++ cross-compiler for Palm-Powered personal digital assistants, primarily for standards-compliance and correct execution of generated code. The technique described here found several hundred bugs, mostly in our in-house code, but also in longstanding high-quality front- and back-end code from Edison Design Group and Apogee Software. It also found eighteen bugs in the GNU C compiler, as well as a bug specific to the Apple version of GCC, a bug specific to the Suse version of GCC, and a dozen bugs in versions of GCC for the ARM processor, several of them critical.