Beyond Static Bias: Adaptive Multi-Fidelity Bandits with Improving Proxies
For bandit practitioners using improving proxies like LLMs, this work provides a principled method to adaptively decide when to use cheap but improving feedback, reducing reliance on expensive high-fidelity evaluations.
This paper introduces adaptive multi-fidelity bandits where low-fidelity proxies improve with use, and proposes TACC algorithm achieving bounded low-fidelity continuation for intermediate arms, reducing cost-weighted regret in synthetic and LLM-based tasks.
As an extension of the classical multi-armed bandit problem, multi-fidelity multi-armed bandits (MF-MAB) enable individual arms to be evaluated using diverse feedback sources that vary in both cost and accuracy. Prior stochastic models typically assume fixed low-to-high fidelity discrepancies, whereas modern proxy sources, such as learning-based simulators and Large Language Models (LLMs), can be improved using additional calibration. We investigate adaptive MF-MAB with improving proxy sources, and focus on the canonical two-fidelity case in which the low-fidelity source becomes more informative with repeated use. To capture this dynamic, we introduce a selected-average mismatch bound that converts dynamic low-fidelity observations into improvement-aware confidence bounds for the high-fidelity target. We propose the Threshold-Based Adaptive Continuation Companion (TACC), an optimistic algorithm that uses a bounded continuation rule to decide when low-fidelity sampling remains cost-effective and when to escalate. We prove an instance-dependent regret bound showing that, for detected intermediate arms, adaptive continuation replaces logarithmic high-fidelity confirmation with bounded low-fidelity continuation. Experiments on synthetic bandits and an LLM-as-a-judge policy-evaluation task examine when continuation improves cost-weighted regret.