PAC-Bayesian Reinforcement Learning Trains Generalizable Policies
This work addresses generalization challenges in reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing PAC-Bayesian and off-policy methods.
The authors tackled the problem of obtaining generalization guarantees in reinforcement learning by deriving a PAC-Bayesian bound that accounts for Markov dependencies, and they demonstrated its utility with a novel algorithm that maintains competitive performance across tasks.
We derive a novel PAC-Bayesian generalization bound for reinforcement learning that explicitly accounts for Markov dependencies in the data, through the chain's mixing time. This contributes to overcoming challenges in obtaining generalization guarantees for reinforcement learning, where the sequential nature of data breaks the independence assumptions underlying classical bounds. Our bound provides non-vacuous certificates for modern off-policy algorithms like Soft Actor-Critic. We demonstrate the bound's practical utility through PB-SAC, a novel algorithm that optimizes the bound during training to guide exploration. Experiments across continuous control tasks show that our approach provides meaningful confidence certificates while maintaining competitive performance.