Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
This addresses a key limitation in reinforcement learning for control tasks, offering a more robust method when models are imperfect, though it appears incremental as it builds on existing differentiation techniques.
The paper tackles the problem of model misspecification in model-based reinforcement learning by proposing an end-to-end approach that directly optimizes expected returns using implicit differentiation, showing benefits over likelihood-based methods in this regime.
The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.