Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
For practitioners deploying LLMs under resource constraints, CARGO provides a practical, adaptable routing method without the need for training or finetuning.
CARGO achieves training-free LLM offloading by using the local model's inference-time agreement across sampled responses to decide when to offload to a stronger cloud model, outperforming other training-free baselines and sometimes surpassing supervised routers.
Local-cloud collaboration is a practical way to deploy large language models under resource constraints, but existing methods often rely on trained routers or collaboration-aware finetuning that tie routing behavior to a particular operating regime. In this work, we show that such training may be unnecessary: the local model's own inference-time agreement across sampled responses already provides a strong signal for deciding when to trust local execution and when to offload to a stronger cloud model. We propose CARGO, a training-free routing framework that estimates this agreement through prompt-varied sampling, applies Bayesian early stopping for sample-efficient uncertainty control, and supports arbitrary target collaboration ratios through lightweight deployment-time calibration. Across diverse reasoning and question-answering tasks, multiple local LLM families and scales, and both pretrained and finetuned local models, CARGO consistently outperforms other training-free baselines and in several settings surpasses supervised learned routers. These results suggest that effective and adaptable local-cloud collaboration can emerge directly from the local model's intrinsic response behavior, without requiring an additional trained router.