Lipschitz Continuity in Model-based Reinforcement Learning
This addresses stability and error control in reinforcement learning models, but it is incremental as it builds on existing Lipschitz continuity concepts.
The paper tackles the problem of multi-step prediction error in model-based reinforcement learning by learning Lipschitz continuous models, resulting in a novel bound on error using the Wasserstein metric and empirical benefits from controlling the Lipschitz constant.
We examine the impact of learning Lipschitz continuous models in the context of model-based reinforcement learning. We provide a novel bound on multi-step prediction error of Lipschitz models where we quantify the error using the Wasserstein metric. We go on to prove an error bound for the value-function estimate arising from Lipschitz models and show that the estimated value function is itself Lipschitz. We conclude with empirical results that show the benefits of controlling the Lipschitz constant of neural-network models.