Empirical Analysis of GPU Frequency Behavior Under ML Workloads
For researchers and practitioners optimizing ML inference on GPUs, this work identifies a previously overlooked source of latency variability that undermines existing prediction methods.
This work reveals that on lower-performance NVIDIA GPUs, operating frequency is strongly influenced by recent workload history within an 80ms window, challenging the assumption of independent kernel latencies used in ML latency-prediction techniques. The findings show that dynamic frequency scaling introduces inter-kernel dependencies, affecting total execution time estimation.
This work presents ongoing research on the frequency scaling behavior of NVIDIA GPUs when executing ML/AI workloads. Our preliminary findings show that, on lower-performance GPUs, the operating frequency is strongly affected by the recent workload history, typically within an 80ms window. This behavior challenges a common assumption underlying several state-of-the-art ML latency-prediction techniques, which treat individual GPU kernel latencies as independent and therefore estimate total execution time by summing isolated per-kernel measurements. Our results indicate that such an assumption does not always hold, as the GPU's dynamic frequency scaling introduces inter-kernel dependencies. We also outline several promising directions for leveraging this observation in future work, including improved latency-prediction models, GPU kernel-reordering strategies, and NAS-driven guidelines for frequency/latency/energy-aware model design.