Toward Explaining Large Language Models in Software Engineering Tasks
This addresses the need for domain-specific explanations in software engineering to improve trust and accountability in high-stakes applications, representing an incremental advance in explainable AI for this domain.
The paper tackles the lack of explainability in Large Language Models for software engineering tasks by introducing FeatureSHAP, a model-agnostic explainability framework that attributes model outputs to high-level input features using Shapley values. Results show it assigns less importance to irrelevant features and produces higher-fidelity explanations than baselines, with a practitioner survey of 37 participants confirming it helps interpret model outputs and make informed decisions.
Recent progress in Large Language Models (LLMs) has substantially advanced the automation of software engineering (SE) tasks, enabling complex activities such as code generation and code summarization. However, the black-box nature of LLMs remains a major barrier to their adoption in high-stakes and safety-critical domains, where explainability and transparency are vital for trust, accountability, and effective human supervision. Despite increasing interest in explainable AI for software engineering, existing methods lack domain-specific explanations aligned with how practitioners reason about SE artifacts. To address this gap, we introduce FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. Based on Shapley values, FeatureSHAP attributes model outputs to high-level input features through systematic input perturbation and task-specific similarity comparisons, while remaining compatible with both open-source and proprietary LLMs. We evaluate FeatureSHAP on two bi-modal SE tasks: code generation and code summarization. The results show that FeatureSHAP assigns less importance to irrelevant input features and produces explanations with higher fidelity than baseline methods. A practitioner survey involving 37 participants shows that FeatureSHAP helps practitioners better interpret model outputs and make more informed decisions. Collectively, FeatureSHAP represents a meaningful step toward practical explainable AI in software engineering. FeatureSHAP is available at https://github.com/deviserlab/FeatureSHAP.