What helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration
This addresses the high cost of CI for software engineers by providing a comprehensive evaluation to guide future research and practice.
The paper performed the first exhaustive comparison of techniques to improve continuous integration (CI), evaluating 14 variants of 10 techniques on 100 real-world projects using 10 metrics to assess cost and time-to-feedback savings, identifying their strengths and weaknesses.
Continuous integration (CI) is a widely used practice in modern software engineering. Unfortunately, it is also an expensive practice - Google and Mozilla estimate their CI systems in millions of dollars. There are a number of techniques and tools designed to or having the potential to save the cost of CI or expand its benefit - reducing time to feedback. However, their benefits in some dimensions may also result in drawbacks in others. They may also be beneficial in other scenarios where they are not designed to help. In this paper, we perform the first exhaustive comparison of techniques to improve CI, evaluating 14 variants of 10 techniques using selection and prioritization strategies on build and test granularity. We evaluate their strengths and weaknesses with 10 different cost and time-tofeedback saving metrics on 100 real-world projects. We analyze the results of all techniques to understand the design decisions that helped different dimensions of benefit. We also synthesized those results to lay out a series of recommendations for the development of future research techniques to advance this area.