How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization
This work addresses a key bottleneck in MBRL for researchers and practitioners, offering an adaptive solution to improve algorithm stability and performance, though it appears incremental as it builds on prior methods.
The paper tackles the challenge of performance deterioration in model-based reinforcement learning due to model shift and bias by proposing a unified optimization objective and fine-tuning process, resulting in state-of-the-art performance on benchmark tasks.
Designing and deriving effective model-based reinforcement learning (MBRL) algorithms with a performance improvement guarantee is challenging, mainly attributed to the high coupling between model learning and policy optimization. Many prior methods that rely on return discrepancy to guide model learning ignore the impacts of model shift, which can lead to performance deterioration due to excessive model updates. Other methods use performance difference bound to explicitly consider model shift. However, these methods rely on a fixed threshold to constrain model shift, resulting in a heavy dependence on the threshold and a lack of adaptability during the training process. In this paper, we theoretically derive an optimization objective that can unify model shift and model bias and then formulate a fine-tuning process. This process adaptively adjusts the model updates to get a performance improvement guarantee while avoiding model overfitting. Based on these, we develop a straightforward algorithm USB-PO (Unified model Shift and model Bias Policy Optimization). Empirical results show that USB-PO achieves state-of-the-art performance on several challenging benchmark tasks.