Agentic Repository Mining: A Multi-Task Evaluation
For software repository mining tasks, this work demonstrates that agent-based approaches can match or outperform static context methods while avoiding context limitations, though the gains are primarily in robustness rather than raw accuracy.
LLM agents that dynamically explore repositories via bash commands achieve competitive accuracy to simple LLMs with pre-engineered context across 4943 classifications on four tasks, with the key advantage of robustness against context-window overflows and independence from artifact size.
Mining software repositories often requires classifying artifacts like commits, reviews, code lines, or entire repositories into categories. Human labeling is expensive and error-prone; limited context frequently leads to misclassifications or uncertainty in labels. We investigate whether LLM agents that dynamically explore repositories through standard bash commands can match the classification quality of simple LLMs that receive pre-engineered context. Across four tasks, eight approach configurations, and 4943 classifications, agents achieve competitive accuracy despite retrieving their own context. The primary advantage is robustness: agents avoid context-window overflows and scale independently of artifact size. A manual diagnosis of 100 cases where approaches disagree with the ground truth reveals specification ambiguities and labels produced under limited context, suggesting that accuracy against such ground truth may underestimate approaches with broader context access.