Bandits with Dynamic Arm-acquisition Costs
This addresses online learning problems with costly resource acquisition, such as in markets with churn, but is incremental as it builds on existing bandit frameworks.
The paper tackles a bandit problem where new arms can be added at a cost from a reservoir with unknown types, and the cost affects the probability of getting an optimal arm. It establishes a necessary condition for sub-linear regret and proposes a UCB-inspired algorithm that is long-run-average optimal when this condition is met.
We consider a bandit problem where at any time, the decision maker can add new arms to her consideration set. A new arm is queried at a cost from an "arm-reservoir" containing finitely many "arm-types," each characterized by a distinct mean reward. The cost of query reflects in a diminishing probability of the returned arm being optimal, unbeknown to the decision maker; this feature encapsulates defining characteristics of a broad class of operations-inspired online learning problems, e.g., those arising in markets with churn, or those involving allocations subject to costly resource acquisition. The decision maker's goal is to maximize her cumulative expected payoffs over a sequence of n pulls, oblivious to the statistical properties as well as types of the queried arms. We study two natural modes of endogeneity in the reservoir distribution, and characterize a necessary condition for achievability of sub-linear regret in the problem. We also discuss a UCB-inspired adaptive algorithm that is long-run-average optimal whenever said condition is satisfied, thereby establishing its tightness.