Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
This work addresses the need for interpretable multi-level visual representations in multimodal LLMs, offering a method to organize concepts hierarchically for better model understanding and control.
Cascaded sparse autoencoders (CSAEs) learn hierarchical visual concepts in multimodal LLMs by training a second-level SAE on the decoder weights of the first-level SAE. Experiments across Qwen3-VL, Gemma-3, and LLaVA show improved hierarchical concept coherence over state-of-the-art SAE baselines, with effective group-level interventions in MLLM outputs.
Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret. Sparse Autoencoders (SAEs) provide a scalable way to decompose dense model activations into sparse, interpretable features. However, existing SAE architectures primarily recover flat feature dictionaries and are less suited for explicit multi-level concept organization. In this paper, we introduce cascaded sparse autoencoders (CSAEs) for learning hierarchical visual concepts in MLLMs. Rather than nesting or stacking SAE sparse activation codes, CSAEs train a second-level SAE directly on the decoder weights of the first-level SAE, treating learned low-level feature directions as inputs for higher-level abstraction. This design enables CSAEs to learn "concepts of concepts" while avoiding drawbacks from the shared-prefix coupling of nesting, Matryoshka-style hierarchies and the bottlenecks of naively stacked SAEs. Experiments across Qwen3-VL, Gemma-3, and LLaVA on multiple visual datasets show that CSAEs improve interpretability in terms of hierarchical concept coherence over state-of-the-art SAE baselines. Results on concept steering further demonstrate that the learned concept groups support effective group-level interventions in MLLM outputs.