PAC-Bayesian Policy Evaluation for Reinforcement Learning
This addresses the need for reliable policy evaluation in reinforcement learning, particularly for transfer learning, by offering a method that is robust to incorrect priors, though it is incremental as it extends PAC-Bayesian theory to a new domain.
The paper tackles the problem of policy evaluation in batch reinforcement learning with function approximation by introducing the first PAC-Bayesian bound, which provides guarantees regardless of prior correctness. Empirical results show it effectively leverages informative priors while ignoring misleading ones, improving robustness in transfer learning scenarios.
Bayesian priors offer a compact yet general means of incorporating domain knowledge into many learning tasks. The correctness of the Bayesian analysis and inference, however, largely depends on accuracy and correctness of these priors. PAC-Bayesian methods overcome this problem by providing bounds that hold regardless of the correctness of the prior distribution. This paper introduces the first PAC-Bayesian bound for the batch reinforcement learning problem with function approximation. We show how this bound can be used to perform model-selection in a transfer learning scenario. Our empirical results confirm that PAC-Bayesian policy evaluation is able to leverage prior distributions when they are informative and, unlike standard Bayesian RL approaches, ignore them when they are misleading.