Pretraining with Token-Level Adaptive Latent Chain-of-Thought
This addresses efficiency and performance challenges in large language model training for AI researchers, though it is incremental as it builds on existing CoT and pretraining methods.
The paper tackles the problem of scaling large language models by increasing per-token computation without expanding parameters, using an internalized latent Chain-of-Thought during pretraining, and results in improved language modeling perplexity and downstream accuracy with fewer training FLOPs.
Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-token computation without expanding parameters, by internalizing latent Chain-of-Thought (CoT) into pretraining. We propose Pretraining with Token-Level Adaptive Latent CoT (adaptive latent CoT), where the model generates a variable-length latent CoT trajectory before emitting each token -- allocating longer trajectories to difficult tokens and shorter (or even zero) trajectories to easy ones. Importantly, this behavior emerges naturally from one-stage pretraining on general text and reduces computation in both training and inference via token-wise adaptive halting. Experiments with Llama architectures show that adaptive latent CoT consistently improves language modeling perplexity and broad downstream accuracy, even with fewer training FLOPs than prior recurrent baselines.