Differential Testing for Variational Analyses: Experience from Developing KConfigReader
This incremental approach benefits developers of variational analysis tools by enhancing testing efficiency and precision in specific domains like software configuration.
The paper tackled the problem of improving the development and precision of variational analysis tools, specifically KConfigReader for translating Linux kernel kconfig models, by applying differential testing to build a large test base and avoid regressions, resulting in KConfigReader becoming likely the most precise tool available.
Differential testing to solve the oracle problem has been applied in many scenarios where multiple supposedly equivalent implementations exist, such as multiple implementations of a C compiler. If the multiple systems disagree on the output for a given test input, we have likely discovered a bug without every having to specify what the expected output is. Research on variational analyses (or variability-aware or family-based analyses) can benefit from similar ideas. The goal of most variational analyses is to perform an analysis, such as type checking or model checking, over a large number of configurations much faster than an existing traditional analysis could by analyzing each configuration separately. Variational analyses are very suitable for differential testing, since the existence nonvariational analysis can provide the oracle for test cases that would otherwise be tedious or difficult to write. In this experience paper, I report how differential testing has helped in developing KConfigReader, a tool for translating the Linux kernel's kconfig model into a propositional formula. Differential testing allows us to quickly build a large test base and incorporate external tests that avoided many regressions during development and made KConfigReader likely the most precise kconfig extraction tool available.