FreeRide: Harvesting Bubbles in Pipeline Parallelism
This addresses inefficiencies in GPU resource utilization for large language model training, offering a practical solution to reduce costs, though it is incremental as it builds on existing pipeline parallelism methods.
The paper tackles the problem of GPU resource underutilization due to bubbles in pipeline parallelism during large language model training, which can account for over 40% of training time, and proposes FreeRide, a system that harvests these bubbles for side tasks, achieving 7.8% average cost savings with about 1% overhead.
The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the main reasons for the underutilization of GPU resources in LLM training. Harvesting these bubbles for GPU side tasks can increase resource utilization and reduce training costs but comes with challenges. First, because bubbles are discontinuous with various shapes, programming side tasks becomes difficult while requiring excessive engineering effort. Second, a side task can compete with pipeline training for GPU resources and incur significant overhead. To address these challenges, we propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide provides programmers with interfaces to implement side tasks easily, manages bubbles and side tasks during pipeline training, and controls access to GPU resources by side tasks to reduce overhead. We demonstrate that FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs while serving model training, graph analytics, and image processing side tasks.