The OoO VLIW JIT Compiler for GPU Inference
This addresses efficiency problems for ML practitioners deploying inference on GPUs, offering a novel method to optimize resource use while meeting latency requirements.
The paper tackles low GPU utilization in ML inference under latency constraints by proposing an OoO VLIW JIT compiler that coalesces and reorders kernels at runtime, achieving a 7.7x improvement in opportunity gap through spatial coalescing.
Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e.g., GPUs, TPUs) point to alarmingly low utilization. As ML inference is increasingly time-bounded by tight latency SLOs, increasing data parallelism is not an option. The need for better efficiency motivates GPU multiplexing. Furthermore, existing GPU programming abstractions force programmers to micro-manage GPU resources in an early-binding, context-free fashion. We propose a VLIW-inspired Out-of-Order (OoO) Just-in-Time (JIT) compiler that coalesces and reorders execution kernels at runtime for throughput-optimal device utilization while satisfying latency SLOs. We quantify the inefficiencies of space-only and time-only multiplexing alternatives and demonstrate an achievable 7.7x opportunity gap through spatial coalescing.