Smooth Sequential Optimisation with Delayed Feedback
This work addresses the challenge of delayed feedback in sequential optimization, which is incremental but offers practical improvements for applications like human-in-the-loop systems.
The paper tackles the problem of unstable sequential learning in multi-armed bandits due to stochastic delays in feedback, proposing a novel adaptation to shrinkage that improves reward estimation stability by over 50% and reduces variability in treatment allocations by up to 3.8x.
Stochastic delays in feedback lead to unstable sequential learning using multi-armed bandits. Recently, empirical Bayesian shrinkage has been shown to improve reward estimation in bandit learning. Here, we propose a novel adaptation to shrinkage that estimates smoothed reward estimates from windowed cumulative inputs, to deal with incomplete knowledge from delayed feedback and non-stationary rewards. Using numerical simulations, we show that this adaptation retains the benefits of shrinkage, and improves the stability of reward estimation by more than 50%. Our proposal reduces variability in treatment allocations to the best arm by up to 3.8x, and improves statistical accuracy - with up to 8% improvement in true positive rates and 37% reduction in false positive rates. Together, these advantages enable control of the trade-off between speed and stability of adaptation, and facilitate human-in-the-loop sequential optimisation.