Budgeted Recommendation with Delayed Feedback
This addresses resource allocation challenges in time-sensitive applications like healthcare, though it appears incremental by extending existing bandit frameworks to incorporate delays and budgets.
The paper tackles the problem of delayed feedback in contextual multi-armed bandits with limited budgets, developing the DORAL policy to optimize resource expenditure in such scenarios, as illustrated by medical supply allocation during COVID-19.
In a conventional contextual multi-armed bandit problem, the feedback (or reward) is immediately observable after an action. Nevertheless, delayed feedback arises in numerous real-life situations and is particularly crucial in time-sensitive applications. The exploration-exploitation dilemma becomes particularly challenging under such conditions, as it couples with the interplay between delays and limited resources. Besides, a limited budget often aggravates the problem by restricting the exploration potential. A motivating example is the distribution of medical supplies at the early stage of COVID-19. The delayed feedback of testing results, thus insufficient information for learning, degraded the efficiency of resource allocation. Motivated by such applications, we study the effect of delayed feedback on constrained contextual bandits. We develop a decision-making policy, delay-oriented resource allocation with learning (DORAL), to optimize the resource expenditure in a contextual multi-armed bandit problem with arm-dependent delayed feedback.