Unlocking Feature Learning in Gated Delta Networks at Scale
It provides principled hyperparameter tuning for efficient sub-quadratic architectures, addressing a key bottleneck in scaling large language models.
The paper derives scaling rules for Gated Delta Networks to enable zero-shot hyperparameter transfer across model widths, demonstrating stable learning-rate transfer under AdamW and SGD, while standard parametrization fails.
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization ($μ$P) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.