Robust Contextual Bandit via the Capped-$\ell_{2}$ norm
This addresses robustness issues in mobile health intervention decision-making, where existing methods are sensitive to outliers, but it is incremental as it builds on standard actor-critic frameworks.
The paper tackles the problem of outliers in mobile health contextual bandit methods by proposing a robust actor-critic approach using the capped-ℓ₂ norm, which dramatically outperforms state-of-the-art methods on datasets with outliers while achieving similar results on clean data.
This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expected reward, which is prone to the existence of outliers. To deal with the issue of outliers, we propose a novel robust actor-critic contextual bandit method for the mHealth intervention. In the critic updating, the capped-$\ell_{2}$ norm is used to measure the approximation error, which prevents outliers from dominating our objective. A set of weights could be achieved from the critic updating. Considering them gives a weighted objective for the actor updating. It provides the badly noised sample in the critic updating with zero weights for the actor updating. As a result, the robustness of both actor-critic updating is enhanced. There is a key parameter in the capped-$\ell_{2}$ norm. We provide a reliable method to properly set it by making use of one of the most fundamental definitions of outliers in statistics. Extensive experiment results demonstrate that our method can achieve almost identical results compared with the state-of-the-art methods on the dataset without outliers and dramatically outperform them on the datasets noised by outliers.