CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performance
This addresses the problem of efficiently co-designing large-scale ML systems and algorithms for industry applications like recommendation models and large language models, representing an incremental improvement in automation.
The paper tackles the challenge of optimizing training performance in large distributed ML systems by proposing CubicML, an automated method that uses an ML model as a proxy to predict performance, achieving effective optimization for models with up to 405 billion parameters at Meta.
Scaling up deep learning models has been proven effective to improve intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of large distributed ML systems and algorithms (to maximize training performance) plays a pivotal role for its success. As it scales, the number of co-design hyper-parameters grows rapidly which brings challenges to feasibly find the optimal setup for system performance maximization. In this paper, we propose CubicML which uses ML to automatically optimize training performance of large distributed ML systems. In CubicML, we use an ML model as a proxy to predict the training performance for search efficiency and performance modeling flexibility. We proved that CubicML can effectively optimize training speed of in-house ads recommendation models with 73 billion parameters and large language models up to 405 billion parameters at Meta.