Predictive Test Selection
This work addresses the challenge of efficient testing in large-scale software development for companies like Facebook, though it is incremental as it builds on existing machine learning techniques.
The paper tackles the problem of high infrastructure costs in continuous integration at Facebook by proposing a predictive test selection strategy that reduces testing costs by half while maintaining over 95% of test failures and 99.9% of faulty changes detection.
Change-based testing is a key component of continuous integration at Facebook. However, a large number of tests coupled with a high rate of changes committed to our monolithic repository make it infeasible to run all potentially-impacted tests on each change. We propose a new predictive test selection strategy which selects a subset of tests to exercise for each change submitted to the continuous integration system. The strategy is learned from a large dataset of historical test outcomes using basic machine learning techniques. Deployed in production, the strategy reduces the total infrastructure cost of testing code changes by a factor of two, while guaranteeing that over 95% of individual test failures and over 99.9% of faulty changes are still reported back to developers. The method we present here also accounts for the non-determinism of test outcomes, also known as test flakiness.