Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
This addresses the sample complexity issue in RL for robotics by enabling transfer learning across tasks with different dimensions, though it is incremental as it builds on prior work by relaxing constraints on state-action space matching.
The paper tackles the problem of transfer learning in deep reinforcement learning when teacher and student tasks have mismatched state and action spaces, proposing a framework that uses embeddings and mutual information loss to enable successful knowledge transfer, demonstrated on simulated robotic locomotion tasks with centipedes.
Deep reinforcement learning (RL) algorithms have achieved great success on a wide variety of sequential decision-making tasks. However, many of these algorithms suffer from high sample complexity when learning from scratch using environmental rewards, due to issues such as credit-assignment and high-variance gradients, among others. Transfer learning, in which knowledge gained on a source task is applied to more efficiently learn a different but related target task, is a promising approach to improve the sample complexity in RL. Prior work has considered using pre-trained teacher policies to enhance the learning of the student policy, albeit with the constraint that the teacher and the student MDPs share the state-space or the action-space. In this paper, we propose a new framework for transfer learning where the teacher and the student can have arbitrarily different state- and action-spaces. To handle this mismatch, we produce embeddings which can systematically extract knowledge from the teacher policy and value networks, and blend it into the student networks. To train the embeddings, we use a task-aligned loss and show that the representations could be enriched further by adding a mutual information loss. Using a set of challenging simulated robotic locomotion tasks involving many-legged centipedes, we demonstrate successful transfer learning in situations when the teacher and student have different state- and action-spaces.