Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning
This work addresses performance degradation and generalization issues in multi-task RL for robotics manipulation, representing an incremental advancement in modular methods.
The paper tackles the problem of negative transfer and limited expressiveness in multi-task reinforcement learning by proposing Contrastive Modules with Temporal Attention (CMTA), which uses contrastive learning and temporal attention to differentiate modules and combine them at a finer granularity, resulting in outperforming individual task learning and achieving substantial improvements over baselines on the Meta-World benchmark.
In the field of multi-task reinforcement learning, the modular principle, which involves specializing functionalities into different modules and combining them appropriately, has been widely adopted as a promising approach to prevent the negative transfer problem that performance degradation due to conflicts between tasks. However, most of the existing multi-task RL methods only combine shared modules at the task level, ignoring that there may be conflicts within the task. In addition, these methods do not take into account that without constraints, some modules may learn similar functions, resulting in restricting the model's expressiveness and generalization capability of modular methods. In this paper, we propose the Contrastive Modules with Temporal Attention(CMTA) method to address these limitations. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level with temporal attention, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL. We conducted the experiment on Meta-World, a multi-task RL benchmark containing various robotics manipulation tasks. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements over the baselines.