Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
For organizations considering GenAI tools, this study highlights a discrepancy between subjective productivity gains and objective activity metrics, suggesting that perceived benefits may not translate to measurable output changes.
This mixed-methods case study in a large public sector organization found that GitHub Copilot users were more active than non-users even before adopting the tool, and no statistically significant changes in commit-based activity were observed after adoption, despite users reporting increased productivity.
This study investigates the real-world impact of the generative AI (GenAI) tool GitHub Copilot on developer activity and perceived productivity. We conducted a mixed-methods case study in NAV IT, a large public sector agile organization. We analyzed 26,317 unique non-merge commits from 703 of NAV IT's GitHub repositories over a two-year period, focusing on commit-based activity metrics from 25 Copilot users and 14 non-users. The analysis was complemented by survey responses on their roles and perceived productivity, as well as 13 interviews. Our analysis of activity metrics revealed that individuals who used Copilot were consistently more active than non-users, even prior to Copilot's introduction. We did not find any statistically significant changes in commit-based activity for Copilot users after they adopted the tool, although minor increases were observed. This suggests a discrepancy between changes in commit-based metrics and the subjective experience of productivity.