A Simulation Study of Bandit Algorithms to Address External Validity of Software Fault Prediction
This addresses the external validity issue in software fault prediction for developers, but it is incremental as it applies existing bandit algorithms to a known problem.
The study tackled the problem of software fault prediction models performing poorly on diverse datasets by applying bandit algorithms to dynamically select models, finding that the Thomson sampling algorithm achieved the best or second-best prediction performance compared to using a single model.
Various software fault prediction models and techniques for building algorithms have been proposed. Many studies have compared and evaluated them to identify the most effective ones. However, in most cases, such models and techniques do not have the best performance on every dataset. This is because there is diversity of software development datasets, and therefore, there is a risk that the selected model or technique shows bad performance on a certain dataset. To avoid selecting a low accuracy model, we apply bandit algorithms to predict faults. Consider a case where player has 100 coins to bet on several slot machines. Ordinary usage of software fault prediction is analogous to the player betting all 100 coins in one slot machine. In contrast, bandit algorithms bet one coin on each machine (i.e., use prediction models) step-by-step to seek the best machine. In the experiment, we developed an artificial dataset that includes 100 modules, 15 of which include faults. Then, we developed various artificial fault prediction models and selected them dynamically using bandit algorithms. The Thomson sampling algorithm showed the best or second-best prediction performance compared with using only one prediction model.