Stabilizing Native Low-Rank LLM Pretraining
This addresses computational and memory challenges in foundation models for AI researchers and practitioners, offering a novel approach to efficient training without prior reliance on full-rank guidance.
The paper tackled the problem of training large language models from scratch using exclusively low-rank factorized weights to reduce computational costs, and demonstrated stable training with a method called Spectron that prevents instability by controlling spectral norms, achieving performance matching dense models and establishing compute-optimal scaling laws.
Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.