The Promise and Reality of Continuous Integration Caching: An Empirical Study of Travis CI Builds
This empirical study identifies practical challenges in CI caching adoption and maintenance for software developers, highlighting incremental insights into real-world usage.
The study analyzed 513,384 Travis CI builds to assess CI caching adoption, finding only 30% of projects use it, with early adopters being more mature and having more dependencies. It revealed that nearly half of non-adopting projects accepted caching when enabled, but cache maintenance is complex, with 27% having stale artifacts and 97% of builds involving uploads, leading to mixed build-time benefits.
Continuous Integration (CI) provides early feedback by automatically building software, but long build durations can hinder developer productivity. CI services use caching to speed up builds by reusing infrequently changing artifacts, yet little is known about how caching is adopted in practice and what challenges it entails. In this paper, we conduct a large-scale empirical study of CI caching in Travis CI, analyzing 513,384 builds from 1,279 GitHub projects. We find that only 30% of projects adopt CI caching, and early adopters are typically more mature, with more dependencies, commits, and longer CI lifespans. To understand non-adoption, we submit pull requests enabling caching in non-adopting projects, and nearly half are accepted or merged. Developer feedback indicates that non- or late adoption mainly results from limited awareness of CI caching support. We further study cache maintenance and identify five common activities, performed by 24% of cache-enabled projects. While one-third of projects see substantial build-time reductions, cache uploads occur in 97% of builds, and 27% of projects contain stale cached artifacts. An analysis of reported caching issues shows developers mainly struggle with corrupted or outdated caches and request broader caching features. Overall, CI caching does not benefit all projects, requires ongoing maintenance, and is more complex in practice than many developers expect.