A Benchmark Generator for Combinatorial Testing
This work addresses a gap for researchers and developers in the CT community by enabling better assessment of tool correctness and performance, though it is incremental as it builds on existing combinatorial problem structures.
The paper tackles the lack of resources for testing Combinatorial Testing (CT) tools by introducing a new benchmark generator that uses structures from other combinatorial problems, and it provides insights on tool selection through an extensive evaluation.
Combinatorial Testing (CT) tools are essential to test properly a wide range of systems (train systems, Graphical User Interfaces (GUIs), autonomous driving systems, etc). While there is an active research community working on developing CT tools, paradoxically little attention has been paid to making available enough resources to test the CT tools themselves. In particular, the set of available benchmarks to asses their correctness, effectiveness and efficiency is rather limited. In this paper, we introduce a new generator of CT benchmarks that essentially borrows the structure contained in the plethora of available Combinatorial Problems from other research communities in order to create meaningful benchmarks. We additionally perform an extensive evaluation of CT tools with these new benchmarks. Thanks to this study we provide some insights on under which circumstances a particular CT tool should be used.