Bandits with Temporal Stochastic Constraints
This addresses the challenge of temporal constraints in bandit problems for domains like advertising and vocational training, representing an incremental advance over prior work on delays and corruptions.
The paper tackles the problem of impairment in stochastic multi-armed bandits, where rewards depend on recent repetitions of actions, by developing two novel algorithms that achieve sublinear regret, with bounds scaling (sub-)linearly with the degree of impairment.
We study the effect of impairment on stochastic multi-armed bandits and develop new ways to mitigate it. Impairment effect is the phenomena where an agent only accrues reward for an action if they have played it at least a few times in the recent past. It is practically motivated by repetition and recency effects in domains such as advertising (here consumer behavior may require repeat actions by advertisers) and vocational training (here actions are complex skills that can only be mastered with repetition to get a payoff). Impairment can be naturally modelled as a temporal constraint on the strategy space, and we provide two novel algorithms that achieve sublinear regret, each working with different assumptions on the impairment effect. We introduce a new notion called bucketing in our algorithm design, and show how it can effectively address impairment as well as a broader class of temporal constraints. Our regret bounds explicitly capture the cost of impairment and show that it scales (sub-)linearly with the degree of impairment. Our work complements recent work on modeling delays and corruptions, and we provide experimental evidence supporting our claims.