Test2Vec: An Execution Trace Embedding for Test Case Prioritization
This addresses the need for more effective test case prioritization in software testing, offering a novel alternative to traditional coverage-based methods, though it is incremental in its application to a specific domain.
The paper tackles the problem of test case prioritization by proposing Test2Vec, an embedding method based on execution traces, which improves the median normalized rank of the first failing test case by 41.80% compared to best alternatives and outperforms code coverage-based approaches by 25.05% to 59.25% in key metrics.
Most automated software testing tasks can benefit from the abstract representation of test cases. Traditionally, this is done by encoding test cases based on their code coverage. Specification-level criteria can replace code coverage to better represent test cases' behavior, but they are often not cost-effective. In this paper, we hypothesize that execution traces of the test cases can be a good alternative to abstract their behavior for automated testing tasks. We propose a novel embedding approach, Test2Vec, that maps test execution traces to a latent space. We evaluate this representation in the test case prioritization (TP) task. Our default TP method is based on the similarity of the embedded vectors to historical failing test vectors. We also study an alternative based on the diversity of test vectors. Finally, we propose a method to decide which TP to choose, for a given test suite. The experiment is based on several real and seeded faults with over a million execution traces. Results show that our proposed TP improves best alternatives by 41.80% in terms of the median normalized rank of the first failing test case (FFR). It outperforms traditional code coverage-based approaches by 25.05% and 59.25% in terms of median APFD and median normalized FFR.