Characterization of Large Language Model Development in the Datacenter
This work addresses inefficiencies in LLM development for researchers and engineers, but it is incremental as it builds on existing systems with specific optimizations.
The paper tackles the problem of efficiently utilizing large-scale cluster resources for developing Large Language Models (LLMs) by presenting a characterization study of a six-month workload trace from a GPU datacenter, identifying challenges like hardware failures and resource imbalances, and introducing system efforts such as fault-tolerant pretraining and decoupled scheduling for evaluation.
Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization.