The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study
This work provides a roadmap for researchers in software engineering by synthesizing emerging research on integrating ML into test generation, though it is incremental as it reviews existing literature rather than proposing new methods.
The study systematically maps 124 publications to characterize how machine learning is applied to automated test generation, finding that ML is used for generating test inputs and oracles across various testing types, with supervised and reinforcement learning being common approaches.
Context: Machine learning (ML) may enable effective automated test generation. Objective: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges. Methods: We perform a systematic mapping on a sample of 124 publications. Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning - often based on neural networks - and reinforcement learning - often based on Q-learning - are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion: Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed - and how they are applied - benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.