Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
This work addresses scalability and efficiency challenges for companies like Tencent that rely on large pre-trained models for products such as WeChat and QQ, though it is incremental as it builds on existing pre-training systems.
The paper tackles the problem of efficiently training large-scale pre-trained Transformer models by introducing Angel-PTM, a system that achieves up to 114.8% improvement in maximum model scale and up to 88.9% increase in training throughput compared to existing systems.
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as WeChat, QQ, and Tencent Advertisement, have been opted in to gain the power of pre-trained models. In this work, we present Angel-PTM, a productive deep learning system designed for pre-training and fine-tuning Transformer models. Angel-PTM can train extremely large-scale models with hierarchical memory efficiently. The key designs of Angel-PTM are the fine-grained memory management via the Page abstraction and a unified scheduling method that coordinate the computations, data movements, and communications. Furthermore, Angel-PTM supports extreme model scaling with SSD storage and implements the lock-free updating mechanism to address the SSD I/O bandwidth bottlenecks. Experimental results demonstrate that Angel-PTM outperforms existing systems by up to 114.8% in terms of maximum model scale as well as up to 88.9% in terms of training throughput. Additionally, experiments on GPT3-175B and T5-MoE-1.2T models utilizing hundreds of GPUs verify the strong scalability of Angel-PTM.