Sparse Autoencoders for Interpretable Medical Image Representation Learning
This addresses the need for interpretable AI in medical imaging for clinicians, though it is incremental as it applies an existing method (SAEs) to a new domain.
The study tackled the problem of opaque latent representations in medical vision foundation models by investigating Sparse Autoencoders (SAEs) to create human-interpretable, sparse features, achieving up to 0.941 R2 reconstruction fidelity and recovering 87.8% of downstream performance with only 10 features (99.4% dimensionality reduction).
Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D image slices from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R2 up to 0.941) and recover up to 87.8% of downstream performance using only 10 features (99.4% dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation. (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: https://github.com/pwesp/sail.