Budget-Adaptive Routing: Skipping the Weak When the Strong Answers Anyway
For edge-cloud inference systems, this work addresses the suboptimality of fixed routing placements under varying offload budgets, offering a practical adaptive solution.
The paper proposes a budget-adaptive routing method for edge-cloud inference that selects between weak-skipping and weak-conditioned placements based on offload budget, achieving up to 19.1 ms latency reduction (30% at rho=0.9) and outperforming the strong model by +1.7 pp mAP at some operating points on PASCAL VOC.
Edge-cloud inference collaborations are often designed with a routing estimator that decides whether to offload each frame from weak models at the edge to stronger models in the cloud. Existing systems place the routing estimator after the weak detector, so the weak forward pass still runs even on frames that are later offloaded. In this paper, we argue that this weak-conditioned design can be suboptimal when the offload budget varies. First, we present a competitive weak-skipping estimator (0.153 GFLOPs, about 29x lighter than the weak detector at 4.49 GFLOPs) that extracts routing signal from raw pixels, outperforming the common after-weak placement weak-conditioned baselines. Second, we show that neither weak-skipping nor weak-conditioned placement dominates across the full operating curve, and we propose budget-adaptive routing, which selects between them by offload budget via two offline-tuned thresholds. On PASCAL VOC, our budget-adaptive router traces the upper accuracy envelope of both fixed placements across the operating range. Our method reduces per-frame latency by up to 19.1 ms (about 30% lower at rho = 0.9). Besides outperforming SOTA methods, it is surprisingly stronger than the strong model (+1.7 pp over the strong model's peak mAP) at some operating points with far less compute. Artifacts are available at https://github.com/ViGeng/bgt-ada