Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent
This work addresses the need for efficient online inference in sequential decision-making for applications like personalized recommendations, though it is incremental as it builds on existing SGD approaches.
The paper tackles the problem of online statistical inference for model parameters in contextual bandits, proposing a framework that updates decision rules via weighted stochastic gradient descent and establishes asymptotic normality, with the estimator significantly improving asymptotic efficiency over previous methods.
With the fast development of big data, learning the optimal decision rule by recursively updating it and making online decisions has been easier than before. We study the online statistical inference of model parameters in a contextual bandit framework of sequential decision-making. We propose a general framework for an online and adaptive data collection environment that can update decision rules via weighted stochastic gradient descent. We allow different weighting schemes of the stochastic gradient and establish the asymptotic normality of the parameter estimator. Our proposed estimator significantly improves the asymptotic efficiency over the previous averaged SGD approach via inverse probability weights. We also conduct an optimality analysis on the weights in a linear regression setting. We provide a Bahadur representation of the proposed estimator and show that the remainder term in the Bahadur representation entails a slower convergence rate compared to classical SGD due to the adaptive data collection.