Partially Observable Contextual Bandits with Linear Payoffs
This addresses a problem in finance where decision-making relies on partially observed market data, representing an incremental advance by adapting existing methods to handle correlated contexts.
The paper tackles the problem of contextual bandits with partially observable and correlated contexts, motivated by finance applications, by proposing an algorithmic pipeline called EMKF-Bandit that integrates system identification, filtering, and bandit algorithms, and shows it achieves sub-linear regret in simulations.
The standard contextual bandit framework assumes fully observable and actionable contexts. In this work, we consider a new bandit setting with partially observable, correlated contexts and linear payoffs, motivated by the applications in finance where decision making is based on market information that typically displays temporal correlation and is not fully observed. We make the following contributions marrying ideas from statistical signal processing with bandits: (i) We propose an algorithmic pipeline named EMKF-Bandit, which integrates system identification, filtering, and classic contextual bandit algorithms into an iterative method alternating between latent parameter estimation and decision making. (ii) We analyze EMKF-Bandit when we select Thompson sampling as the bandit algorithm and show that it incurs a sub-linear regret under conditions on filtering. (iii) We conduct numerical simulations that demonstrate the benefits and practical applicability of the proposed pipeline.