Towards a Benchmark and a Comparison Framework for Combinatorial Interaction Testing of Software Product Lines
This work is incremental, aiming to improve testing practices for software product lines by providing tools for better evaluation of CIT methods.
The paper addresses the lack of systematic comparison among Combinatorial Interaction Testing (CIT) techniques for Software Product Lines (SPLs) by proposing a benchmark of 19 feature models and analyzing an existing comparison framework to identify shortcomings and alternatives.
As Software Product Lines (SPLs) are becoming a more pervasive development practice, their effective testing is becoming a more important concern. In the past few years many SPL testing approaches have been proposed, among them, are those that support Combinatorial Interaction Testing (CIT) whose premise is to select a group of products where faults, due to feature interactions, are more likely to occur. Many CIT techniques for SPL testing have been put forward; however, no systematic and comprehensive comparison among them has been performed. To achieve such goal two items are important: a common benchmark of feature models, and an adequate comparison framework. In this research-in-progress paper, we propose 19 feature models as the base of a benchmark, which we apply to three different techniques in order to analyze the comparison framework proposed by Perrouin et al. We identify the shortcomings of this framework and elaborate alternatives for further study.