SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning
This addresses the challenge of cost-efficient autoscaling for ML serving systems, which is incremental as it builds on existing autoscaling and RL methods with novel components like in-context learning and GPU rate control.
The paper tackles the problem of autoscaling multi-stage ML inference pipelines by introducing SAIR, a framework that uses an in-context reinforcement learning controller with an LLM, achieving up to 50% improvement in P99 latency and up to 97% reduction in effective resource cost on four pipelines under three workload patterns.
Multi-stage ML inference pipelines are difficult to autoscale due to heterogeneous resources, cross-stage coupling, and dynamic bottleneck migration. We present SAIR, an autoscaling framework that uses an LLM as an in-context reinforcement learning controller, improving its policy online from reward-labeled interaction histories without gradient updates. SAIR combines Pareto-dominance reward shaping with a provable separation margin, surprisal-guided experience retrieval for context efficiency, and fine-grained GPU rate control via user-space CUDA interception. We provide regret analysis decomposing error into retrieval coverage and LLM selection components. On four ML serving pipelines under three workload patterns, SAIR achieves the best or tied-best P99 latency and effective resource cost among deployed baselines, improving P99 by up to 50% and reducing effective cost by up to 97% (under GPU rate-control assumptions), with 86% bottleneck detection accuracy and no offline training.