Transfer RL across Observation Feature Spaces via Model-Based Regularization
This addresses a practical inefficiency for RL developers and users facing observation space changes, though it is incremental as it builds on existing transfer learning concepts.
The paper tackles the problem of inefficient retraining in reinforcement learning when the observation space changes, by proposing a model-based regularization algorithm that transfers latent-space dynamics from source to target tasks, resulting in significant improvements in learning efficiency and stability.
In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task.