Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
For practitioners seeking efficient long-context models, this method solves the brittle initialization problem in converting Transformers to hybrid linear attention, enabling faster and more effective distillation.
Taylor-Calibrate provides a principled initialization for converting pretrained Transformers into hybrid linear attention models (Gated DeltaNet), achieving up to 88x improvement in zero-shot performance and reducing training tokens by 4.9x–9.2x to reach matched recovery targets.
Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization rather than learning the remaining teacher behavior. We propose Taylor-Calibrate, a lightweight initialization method for hybrid GDN students. The method uses Taylor-guided teacher attention statistics to set the value projection, memory timescale, write gates, and output gate, then applies a short per-layer alignment step to match each converted layer to the teacher output. Across four teacher settings and three retained-layer policies, Taylor-Calibrate gives substantially stronger zero-shot students, with up to an 88x improvement in a representative ablation, and reaches matched recovery targets with 4.9x--9.2x fewer training tokens than naive conversion.