From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents
This work addresses the opacity of LLM agents' decision-making for researchers and developers, offering a method to enhance trustworthiness in autonomous systems, though it is incremental as it builds on existing interpretability techniques.
The paper tackles the problem of interpreting the internal mechanisms of LLM agents during sequential tasks by developing a framework that uses conformal prediction and linear probes to identify temporal concepts in activation spaces, showing these concepts are linearly separable and can be used for early failure detection and performance improvement in simulated environments like ScienceWorld and AlfWorld.
Large Language Models (LLMs) are increasingly deployed as autonomous agents capable of reasoning, planning, and acting within interactive environments. Despite their growing capability to perform multi-step reasoning and decision-making tasks, internal mechanisms guiding their sequential behavior remain opaque. This paper presents a framework for interpreting the temporal evolution of concepts in LLM agents through a step-wise conformal lens. We introduce the conformal interpretability framework for temporal tasks, which combines step-wise reward modeling with conformal prediction to statistically label model's internal representation at each step as successful or failing. Linear probes are then trained on these representations to identify directions of temporal concepts - latent directions in the model's activation space that correspond to consistent notions of success, failure or reasoning drift. Experimental results on two simulated interactive environments, namely ScienceWorld and AlfWorld, demonstrate that these temporal concepts are linearly separable, revealing interpretable structures aligned with task success. We further show preliminary results on improving an LLM agent's performance by leveraging the proposed framework for steering the identified successful directions inside the model. The proposed approach, thus, offers a principled method for early failure detection as well as intervention in LLM-based agents, paving the path towards trustworthy autonomous language models in complex interactive settings.