Model-based adaptation for sample efficient transfer in reinforcement learning control of parameter-varying systems
This work addresses sample efficiency for reinforcement learning in control systems, offering a method to improve transfer learning with guaranteed positive transfer, though it appears incremental as it builds on existing model-based and transfer learning ideas.
The paper tackles the sample efficiency problem in reinforcement learning for parameter-varying systems by proposing a model-based transformation to ensure positive transfer, achieving comparable performance to linear-quadratic-regulators and model-predictive control in three benchmark cases when an accurate model is known.
In this paper, we leverage ideas from model-based control to address the sample efficiency problem of reinforcement learning (RL) algorithms. Accelerating learning is an active field of RL highly relevant in the context of time-varying systems. Traditional transfer learning methods propose to use prior knowledge of the system behavior to devise a gradual or immediate data-driven transformation of the control policy obtained through RL. Such transformation is usually computed by estimating the performance of previous control policies based on measurements recently collected from the system. However, such retrospective measures have debatable utility with no guarantees of positive transfer in most cases. Instead, we propose a model-based transformation, such that when actions from a control policy are applied to the target system, a positive transfer is achieved. The transformation can be used as an initialization for the reinforcement learning process to converge to a new optimum. We validate the performance of our approach through four benchmark examples. We demonstrate that our approach is more sample-efficient than fine-tuning with reinforcement learning alone and achieves comparable performance to linear-quadratic-regulators and model-predictive control when an accurate linear model is known in the three cases. If an accurate model is not known, we empirically show that the proposed approach still guarantees positive transfer with jump-start improvement.