Capacity-Constrained Online Learning with Delays: Scheduling Frameworks and Regret Trade-offs
This work addresses the problem of robust online learning under delayed feedback for machine learning practitioners, showing it can be achieved with modest tracking capacity, though it is incremental as it builds on classical delayed online learning frameworks.
The paper tackles online learning with delayed feedback under a capacity constraint limiting how many past rounds can be tracked, establishing matching upper and lower bounds on regret to characterize the optimal capacity needed to match classical minimax rates. For bandits with K actions and total delay D over T rounds, under clairvoyance and capacity C = Ω(log(T)), they achieve regret Θ̃(√(TK + DK/C + D log(K))), and for full-information feedback, Θ̃(√((D+T) log(K))).
We study online learning with oblivious losses and delays under a novel ``capacity constraint'' that limits how many past rounds can be tracked simultaneously for delayed feedback. Under ``clairvoyance'' (i.e., delay durations are revealed upfront each round) and/or ``preemptibility'' (i.e., we can stop tracking previously chosen round feedback), we establish matching upper and lower bounds (up to logarithmic terms) on achievable regret, characterizing the ``optimal capacity'' needed to match the minimax rates of classical delayed online learning, which implicitly assume unlimited capacity. Our algorithms achieve minimax-optimal regret across all capacity levels, with performance gracefully degrading under suboptimal capacity. For $K$ actions and total delay $D$ over $T$ rounds, under clairvoyance and assuming capacity $C = Ω(\log(T))$, we achieve regret $\widetildeΘ(\sqrt{TK + DK/C + D\log(K)})$ for bandits and $\widetildeΘ(\sqrt{(D+T)\log(K)})$ for full-information feedback. When replacing clairvoyance with preemptibility, we require a known maximum delay bound $d_{\max}$, adding ${\widetilde{O}(d_{\max})}$ to the regret. For fixed delays $d$ (i.e., $D=Td$), the minimax regret is $Θ(\sqrt{TK(1+d/C)+Td\log(K)})$ and the optimal capacity is $Θ(\min\{K/\log(K),d\})$ in the bandit setting, while in the full-information feedback setting, the minimax regret is $Θ(\sqrt{T(d+1)\log(K)})$ and the optimal capacity is $Θ(1)$. For round-dependent and fixed delays, our upper bounds are achieved using novel preemptive and non-preemptive scheduling policies, based on Pareto-distributed proxy delays, and batching techniques, respectively. Crucially, our work unifies delayed bandits, label-efficient learning, and online scheduling frameworks, demonstrating that robust online learning under delayed feedback is possible with surprisingly modest tracking capacity.