Logarithmic Smoothing for Adaptive PAC-Bayesian Off-Policy Learning
This work addresses the problem of improving policy learning efficiency for reinforcement learning practitioners by enabling adaptive data collection, though it builds incrementally on existing PAC-Bayesian frameworks.
The paper tackles adaptive off-policy learning by extending PAC-Bayesian learning with Logarithmic Smoothing to iteratively refine policies, achieving faster convergence rates and outperforming static methods when intermediate deployments are allowed.
Off-policy learning serves as the primary framework for learning optimal policies from logged interactions collected under a static behavior policy. In this work, we investigate the more practical and flexible setting of adaptive off-policy learning, where policies are iteratively refined and re-deployed to collect higher-quality data. Building on the success of PAC-Bayesian learning with Logarithmic Smoothing (LS) in static settings, we extend this framework to the adaptive scenario using tools from online PAC-Bayesian theory. Furthermore, we demonstrate that a principled adjustment to the LS estimator naturally accommodates multiple rounds of deployment and yields faster convergence rates under mild conditions. Our method matches the performance of leading offline approaches in static settings, and significantly outperforms them when intermediate policy deployments are allowed. Empirical evaluations across diverse scenarios highlight both the advantages of adaptive data collection and the strength of the PAC-Bayesian formulation.