Beyond Dense States: Elevating Sparse Transcoders to Active Operators for Latent Reasoning
This work addresses the interpretability issue in latent reasoning for AI systems, representing an incremental advancement by integrating sparse representations into active operators.
The paper tackled the problem of interpretability and control in latent reasoning by proposing LSTR, a framework that uses sparse semantic transitions, which preserved reasoning accuracy and compression efficiency while substantially improving interpretability over dense baselines.
Latent reasoning compresses the chain-of-thought (CoT) into continuous hidden states, yet existing methods rely on dense latent transitions that remain difficult to interpret and control. Meanwhile, sparse representation models uncover human-interpretable semantic features but remain largely confined to post-hoc analysis. We reconcile this tension by proposing LSTR (Latent Sparse Transcoder Reasoning), a latent reasoning framework that elevates functional sparse transcoders into active reasoning operators to perform multi-step computation through sparse semantic transitions. At its core, LSTR employs a Latent Transition Transcoder (LTT) with a residual skip architecture that decouples linear manifold transport from sparse semantic updates, enabling controllable semantic resolution via explicit sparsity constraints. Extensive experiments show that LSTR preserves reasoning accuracy and compression efficiency while substantially improving interpretability over dense latent baselines. Causal interventions and trajectory analyses further demonstrate that these sparse features act as both interpretable and causally effective operators in the reasoning process.