Robust Offline Policy Learning with Observational Data from Multiple Sources
This addresses robust policy learning for decision-making systems using diverse observational data, representing an incremental advance in offline reinforcement learning methodology.
The paper tackles the problem of learning personalized decision policies from observational bandit data across multiple heterogeneous sources to ensure robust generalization. It proposes a minimax regret optimization approach that achieves minimal worst-case mixture regret with a vanishing rate dependent on total data.
We consider the problem of using observational bandit feedback data from multiple heterogeneous data sources to learn a personalized decision policy that robustly generalizes across diverse target settings. To achieve this, we propose a minimax regret optimization objective to ensure uniformly low regret under general mixtures of the source distributions. We develop a policy learning algorithm tailored to this objective, combining doubly robust offline policy evaluation techniques and no-regret learning algorithms for minimax optimization. Our regret analysis shows that this approach achieves the minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources. Our analysis, extensions, and experimental results demonstrate the benefits of this approach for learning robust decision policies from multiple data sources.