Less is More: Efficient Weight Farcasting with 1-Layer Neural Network
This addresses the problem of high training costs for large models, offering a streamlined method that is incremental in its approach.
The paper tackles the computational challenge of training large-scale deep neural networks by introducing a framework that uses long-term time series forecasting with only initial and final weight values, achieving improved forecasting accuracy and computational efficiency with minimal overhead, as demonstrated on DistilBERT and synthetic sequences.
Addressing the computational challenges inherent in training large-scale deep neural networks remains a critical endeavor in contemporary machine learning research. While previous efforts have focused on enhancing training efficiency through techniques such as gradient descent with momentum, learning rate scheduling, and weight regularization, the demand for further innovation continues to burgeon as model sizes keep expanding. In this study, we introduce a novel framework which diverges from conventional approaches by leveraging long-term time series forecasting techniques. Our method capitalizes solely on initial and final weight values, offering a streamlined alternative for complex model architectures. We also introduce a novel regularizer that is tailored to enhance the forecasting performance of our approach. Empirical evaluations conducted on synthetic weight sequences and real-world deep learning architectures, including the prominent large language model DistilBERT, demonstrate the superiority of our method in terms of forecasting accuracy and computational efficiency. Notably, our framework showcases improved performance while requiring minimal additional computational overhead, thus presenting a promising avenue for accelerating the training process across diverse tasks and architectures.