Deep Contextual Multi-armed Bandits
This addresses the exploration-exploitation trade-off in industrial applications like marketing optimization, offering an incremental improvement over prior methods by enabling automatic adjustment of exploration levels.
The paper tackles the problem of contextual multi-armed bandits by proposing a deep learning framework that combines non-linear modeling with principled exploration using Thompson sampling and learned dropout rates, resulting in substantially reduced regret on tasks like the UCI Mushroom and Casino Parity tasks compared to existing methods.
Contextual multi-armed bandit problems arise frequently in important industrial applications. Existing solutions model the context either linearly, which enables uncertainty driven (principled) exploration, or non-linearly, by using epsilon-greedy exploration policies. Here we present a deep learning framework for contextual multi-armed bandits that is both non-linear and enables principled exploration at the same time. We tackle the exploration vs. exploitation trade-off through Thompson sampling by exploiting the connection between inference time dropout and sampling from the posterior over the weights of a Bayesian neural network. In order to adjust the level of exploration automatically as more data is made available to the model, the dropout rate is learned rather than considered a hyperparameter. We demonstrate that our approach substantially reduces regret on two tasks (the UCI Mushroom task and the Casino Parity task) when compared to 1) non-contextual bandits, 2) epsilon-greedy deep contextual bandits, and 3) fixed dropout rate deep contextual bandits. Our approach is currently being applied to marketing optimization problems at HubSpot.