The Impact of Documentation on Test Engagement in Pull Requests in OSS
For open-source project maintainers, this provides preliminary evidence that proactive documentation on testing may encourage contributors to write tests, though the effect is weak and correlational.
This study investigates whether documentation on testing in open-source software repositories correlates with contributor testing behavior, measured by the Test Engagement Ratio (TER). Analyzing 160 repositories, they found a weak but significant positive correlation (ρ=0.36), which strengthens in high-activity repositories (ρ=0.44).
Automated testing is crucial for maintaining open-source software quality. However, motivating contributors to include tests for code changes remains a challenge. While existing interventions, such as code coverage metrics and reviewer feedback, are often reactive and applied only after a pull request is opened, this study investigates whether documentation on testing can serve as a proactive measure to encourage testing behavior. In this work, we investigate the relationship between documentation on testing and contributor testing behavior. We introduce the Test Engagement Ratio (TER) to help understand testing frequency. Using data from 160 OSS repositories, we analyze the relationship between documentation comprehensiveness and TER. Our results show a weak but statistically significant positive correlation ($ρ=0.36$, $p<0.001$), which strengthens to a moderate relationship ($ρ=0.44$) in repositories with higher pull request activity. Documentation categories such as How to Run Tests and How to Write Tests show the strongest correlation with testing engagement. Furthermore, TER is found to be moderately correlated ($ρ=0.52$, $p<0.001$) with Test Code Ratio, providing preliminary evidence of its validity. Our findings suggest that documentation on testing may be associated with increased testing engagement. Future work will explore causality, documentation quality at a granular level, and cross-repository exposure effects.