Efficient Pre-Training of LLMs through Truncated SVD Layers
For researchers and engineers training large language models, TSVD offers a practical method to reduce pretraining costs without sacrificing performance.
TSVD reduces LLM pretraining compute by using truncated SVD layers with adaptive rank selection and enforced orthonormality, matching or exceeding full-parameter baseline performance while significantly lowering computational cost.
The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining.