Automated Evolutionary Optimization for Resource-Efficient Neural Network Training
This addresses the need for scalable and resource-efficient models, particularly for applications like financial event sequences, images, and time-series, but it is incremental as it builds on existing AutoML and optimization techniques.
The paper tackles the problem of resource-efficient neural network training by developing an AutoML framework called PETRA, which uses evolutionary optimization to reduce model size by up to 75% and latency by up to 33% while maintaining performance.
There are many critical challenges in optimizing neural network models, including distributed computing, compression techniques, and efficient training, regardless of their application to specific tasks. Solving such problems is crucial because the need for scalable and resource-efficient models is increasing. To address these challenges, we have developed a new automated machine learning (AutoML) framework, Parameter Efficient Training with Robust Automation (PETRA). It applies evolutionary optimization to model architecture and training strategy. PETRA includes pruning, quantization, and loss regularization. Experimental studies on real-world data with financial event sequences, as well as image and time-series -- benchmarks, demonstrate PETRA's ability to improve neural model performance and scalability -- namely, a significant decrease in model size (up to 75%) and latency (up to 33%), and an increase in throughput (by 13%) without noticeable degradation in the target metric.