RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
This work provides critical insights into performance bottlenecks for researchers and practitioners developing and deploying deep reinforcement learning systems, particularly concerning ML backend choices and algorithm types.
This paper investigates system-level bottlenecks in deep reinforcement learning (RL) workloads, finding them less GPU-bound than supervised learning. They developed RL-Scope, a cross-stack profiler, to analyze training time, revealing a 2.3x runtime difference between PyTorch and TensorFlow implementations due to abstraction issues and showing on-policy algorithms are at least 3.5x more simulation-bound than off-policy ones.
Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL workloads are poorly understood; we observe fundamental structural differences in RL workloads that make them inherently less GPU-bound than supervised learning (SL). To explain where training time is spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Using RL-Scope, we survey RL workloads across its major dimensions including ML backend, RL algorithm, and simulator. For ML backends, we explain a $2.3\times$ difference in runtime between equivalent PyTorch and TensorFlow algorithm implementations, and identify a bottleneck rooted in overly abstracted algorithm implementations. For RL algorithms and simulators, we show that on-policy algorithms are at least $3.5\times$ more simulation-bound than off-policy algorithms. Finally, we profile a scale-up workload and demonstrate that GPU utilization metrics reported by commonly used tools dramatically inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope .