Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning
This work addresses interpretability for LLM-based multi-agent systems, which is crucial for ensuring trustworthy AI behavior, but it is incremental as it builds on existing SAE and LLM methods for data-centric analysis.
The paper tackles the problem of understanding behavior changes in LLM-based multi-agent reinforcement learning during training by applying sparse autoencoders (SAEs) and LLM-summarizer methods to analyze training runs in Full-Press Diplomacy, discovering fine-grained and strategic behaviors and validating that 90% of discovered SAE meta-features are significant, with some hypotheses improving an untrained agent's score by +14.2%.
Large language models (LLMs) are increasingly trained in complex Reinforcement Learning, multi-agent environments, making it difficult to understand how behavior changes over training. Sparse Autoencoders (SAEs) have recently shown to be useful for data-centric interpretability. In this work, we analyze large-scale reinforcement learning training runs from the sophisticated environment of Full-Press Diplomacy by applying pretrained SAEs, alongside LLM-summarizer methods. We introduce Meta-Autointerp, a method for grouping SAE features into interpretable hypotheses about training dynamics. We discover fine-grained behaviors including role-playing patterns, degenerate outputs, language switching, alongside high-level strategic behaviors and environment-specific bugs. Through automated evaluation, we validate that 90% of discovered SAE Meta-Features are significant, and find a surprising reward hacking behavior. However, through two user studies, we find that even subjectively interesting and seemingly helpful SAE features may be worse than useless to humans, along with most LLM generated hypotheses. However, a subset of SAE-derived hypotheses are predictively useful for downstream tasks. We further provide validation by augmenting an untrained agent's system prompt, improving the score by +14.2%. Overall, we show that SAEs and LLM-summarizer provide complementary views into agent behavior, and together our framework forms a practical starting point for future data-centric interpretability work on ensuring trustworthy LLM behavior throughout training.