Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
For RL theorists, it provides the first theoretical guarantees for KL-regularized RL under model misspecification, addressing a gap in existing realizability-based analyses.
This work extends KL-regularized reinforcement learning to misspecified models, providing high-probability regret guarantees with explicit misspecification terms that recover the realizable case as a special case.
We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.