Meta-Learning Hypothesis Spaces for Sequential Decision-making
This addresses the challenge of kernel misspecification in sequential decision-making for researchers and practitioners, offering a method to improve safety and performance, though it is incremental as it builds on existing kernel combination frameworks.
The paper tackles the problem of designing reliable confidence sets for sequential decision-making by meta-learning a kernel from offline data, ensuring that with sufficient data, the estimated kernel yields confidence sets as tight as those from the true kernel, with demonstrated competitive regret bounds in kernelized bandit tasks.
Obtaining reliable, adaptive confidence sets for prediction functions (hypotheses) is a central challenge in sequential decision-making tasks, such as bandits and model-based reinforcement learning. These confidence sets typically rely on prior assumptions on the hypothesis space, e.g., the known kernel of a Reproducing Kernel Hilbert Space (RKHS). Hand-designing such kernels is error prone, and misspecification may lead to poor or unsafe performance. In this work, we propose to meta-learn a kernel from offline data (Meta-KeL). For the case where the unknown kernel is a combination of known base kernels, we develop an estimator based on structured sparsity. Under mild conditions, we guarantee that our estimated RKHS yields valid confidence sets that, with increasing amounts of offline data, become as tight as those given the true unknown kernel. We demonstrate our approach on the kernelized bandit problem (a.k.a.~Bayesian optimization), where we establish regret bounds competitive with those given the true kernel. We also empirically evaluate the effectiveness of our approach on a Bayesian optimization task.