Do Natural Language Descriptions of Model Activations Convey Privileged Information?
This work addresses the interpretability of LLMs for researchers and practitioners, highlighting potential pitfalls in current verbalization approaches.
The study critically evaluated whether natural language descriptions of LLM activations provide privileged insights into model internals, finding that verbalizations often reflect the verbalizer LLM's knowledge rather than the target model's and that existing benchmarks may not effectively assess these methods.
Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually provide privileged knowledge about the internal workings of the target model, or do they merely convey information about its inputs? We critically evaluate popular verbalization methods across datasets used in prior work and find that they can succeed at benchmarks without any access to target model internals, suggesting that these datasets may not be ideal for evaluating verbalization methods. We then run controlled experiments which reveal that verbalizations often reflect the parametric knowledge of the verbalizer LLM which generated them, rather than the knowledge of the target LLM whose activations are decoded. Taken together, our results indicate a need for targeted benchmarks and experimental controls to rigorously assess whether verbalization methods provide meaningful insights into the operations of LLMs.