Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders
This work addresses the critical need for safe deployment of AI-suggested code in software development by providing mechanistic insights into LLMs' internal correctness mechanisms, though it is incremental in applying existing interpretability methods to a specific domain.
The researchers tackled the problem of understanding how large language models (LLMs) internally represent code correctness by applying sparse autoencoders to decompose their representations, finding that identified directions reliably predict incorrect code and that successful code generation depends on attending to test cases rather than problem descriptions.
As Large Language Models become integral to software development, with substantial portions of AI-suggested code entering production, understanding their internal correctness mechanisms becomes critical for safe deployment. We apply sparse autoencoders to decompose LLM representations, identifying directions that correspond to code correctness. We select predictor directions using t-statistics and steering directions through separation scores from base model representations, then analyze their mechanistic properties through steering, attention analysis, and weight orthogonalization. We find that code correctness directions in LLMs reliably predict incorrect code, while correction capabilities, though statistically significant, involve tradeoffs between fixing errors and preserving correct code. Mechanistically, successful code generation depends on attending to test cases rather than problem descriptions. Moreover, directions identified in base models retain their effectiveness after instruction-tuning, suggesting code correctness mechanisms learned during pre-training are repurposed during fine-tuning. Our mechanistic insights suggest three practical applications: prompting strategies should prioritize test examples over elaborate problem descriptions, predictor directions can serve as error alarms for developer review, and these same predictors can guide selective steering, intervening only when errors are anticipated to prevent the code corruption from constant steering.