Feature Starvation as Geometric Instability in Sparse Autoencoders
For researchers using sparse autoencoders to interpret LLMs, this work provides a principled, fully differentiable solution to the practical problem of dead neurons and shrinkage bias.
The paper identifies feature starvation in sparse autoencoders as a geometric instability caused by ℓ1 regularization, and proposes adaptive elastic net SAEs (AEN-SAEs) that use ℓ2 regularization and adaptive ℓ1 reweighting to eliminate starvation and shrinkage bias. AEN-SAEs achieve competitive reconstruction without heuristic resampling across models like Pythia 70M and Llama 3.1 8B.
Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization-geometric pathology of overcomplete dictionaries: the $\ell_1$-induced sparse coding map is unstable and fundamentally misaligned with shallow, amortized encoders. To address this structural instability, we introduce adaptive elastic net SAEs (AEN-SAEs), a fully differentiable architecture grounded in classical sparse regression. AEN-SAEs combine an $\ell_2$ structural term that enforces strong convexity and Lipschitz stability with adaptive $\ell_1$ reweighting that eliminates shrinkage bias and suppresses spurious features, thereby jointly controlling the curvature and interaction structure of the induced polyhedral geometry. Theoretically, we show that AEN-SAEs yield a Lipschitz-continuous sparse coding map and recover the global feature support under mild assumptions. Empirically, across synthetic settings and LLMs (Pythia 70M, Llama 3.1 8B), AEN-SAEs mitigate feature starvation without auxiliary heuristics while maintaining competitive reconstruction abilities.