Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
For practitioners seeking efficient improvements to transformer-based language models, LRT offers a low-cost way to enhance performance without architectural overhauls.
The paper introduces Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a previous token's hidden state as recurrent memory, improving language modeling loss and in-context learning under matched effective compute with only 0.3% parameter overhead.
We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this source state is already computed during ordinary decoding, LRT adds a cross-layer recurrent latent pathway across positions without inserting pause tokens or extra depth loops, and the standard attention mechanism and KV-cache interface are preserved. To pretrain this recurrence at scale without sequentially unrolling the transformer, we introduce interleaved parallel training: a single full-sequence initialization forward pass builds a shared buffer; then disjoint position subsets are refined in parallel and written back, so that all tokens receive recurrent-memory-aware supervision at roughly 2 times baseline compute. Across nanochat style backbones and a wide range of tokens-per-parameter budgets, LRT improves both language-modeling loss and in-context learning under matched effective compute while adding as little as 0.3% parameters.