Thompson Sampling in Dynamic Systems for Contextual Bandit Problems
This addresses contextual bandit problems for researchers in reinforcement learning and decision-making, but it appears incremental as it builds on existing Thompson Sampling methods with modifications for dynamic systems.
The paper tackles contextual bandit problems in time-varying dynamic systems by proposing Thompson Sampling with approximate posterior inference via Laplace Approximation and discount decays on previous samples. The result is an adaptive trade-off between exploration and exploitation that accounts for system dynamics, though no concrete performance numbers are provided.
We consider the multiarm bandit problems in the timevarying dynamic system for rich structural features. For the nonlinear dynamic model, we propose the approximate inference for the posterior distributions based on Laplace Approximation. For the context bandit problems, Thompson Sampling is adopted based on the underlying posterior distributions of the parameters. More specifically, we introduce the discount decays on the previous samples impact and analyze the different decay rates with the underlying sample dynamics. Consequently, the exploration and exploitation is adaptively tradeoff according to the dynamics in the system.