Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes
This work highlights hardware-specific performance anomalies for practitioners using Apple MPS for transformer inference, cautioning against reliance on aggregated benchmarks.
The paper identifies non-monotonic latency behavior in Apple MPS backend during autoregressive inference, with latency spikes up to 21x within specific decoding-budget intervals, while CPU and NVIDIA T4 show smooth scaling.
Autoregressive inference is typically assumed to scale predictably with decoding length, and key-value (KV) caching is widely regarded as a universally beneficial optimization for accelerating decoding. In this work, we identify unexpected non-monotonic latency behavior in the Apple MPS backend, where latency changes abruptly across nearby decoding configurations. Using transformer models from multiple families (GPT-2, BLOOM, and OPT), we observe latency spikes of up to 21x within specific decoding-budget intervals, followed by recovery at neighboring configurations. Controlled experiments show that these anomalies are not explained by memory pressure or prefill cost, but are instead consistent with backend execution dynamics, while CPU and NVIDIA T4 (CUDA) exhibit smooth monotonic scaling under identical conditions. Our findings highlight the importance of hardware-aware evaluation for autoregressive inference and caution against relying on aggregated decoding-budget benchmarks, as performance can vary discontinuously across nearby configurations.