Modular Networks Prevent Catastrophic Interference in Model-Based Multi-Task Reinforcement Learning
This addresses the issue of performance degradation in model-based multi-task RL for AI systems, but it is incremental as it builds on known modular approaches to mitigate interference.
The paper tackled the problem of catastrophic interference in model-based multi-task reinforcement learning when using a single shared dynamics model, finding that it leads to task confusion and reduced performance. As a remedy, using modular networks with isolated sub-networks for each task notably improved performance while maintaining the same parameter count, as demonstrated on gridworld and VizDoom experiments.
In a multi-task reinforcement learning setting, the learner commonly benefits from training on multiple related tasks by exploiting similarities among them. At the same time, the trained agent is able to solve a wider range of different problems. While this effect is well documented for model-free multi-task methods, we demonstrate a detrimental effect when using a single learned dynamics model for multiple tasks. Thus, we address the fundamental question of whether model-based multi-task reinforcement learning benefits from shared dynamics models in a similar way model-free methods do from shared policy networks. Using a single dynamics model, we see clear evidence of task confusion and reduced performance. As a remedy, enforcing an internal structure for the learned dynamics model by training isolated sub-networks for each task notably improves performance while using the same amount of parameters. We illustrate our findings by comparing both methods on a simple gridworld and a more complex vizdoom multi-task experiment.