Simultaneously Evolving Deep Reinforcement Learning Models using Multifactorial Optimization
This work addresses a specific bottleneck in reinforcement learning for researchers and practitioners by automating knowledge transfer and policy learning, though it appears incremental as it combines existing techniques like meta-heuristic optimization and transfer learning.
The paper tackles the challenge of Deep Q Learning models struggling with convergence due to exploration issues or sparse rewards by proposing a Multifactorial Optimization framework that simultaneously evolves multiple models for interrelated reinforcement learning tasks, resulting in improved convergence speed and policy quality compared to traditional transfer learning methods.
In recent years, Multifactorial Optimization (MFO) has gained a notable momentum in the research community. MFO is known for its inherent capability to efficiently address multiple optimization tasks at the same time, while transferring information among such tasks to improve their convergence speed. On the other hand, the quantum leap made by Deep Q Learning (DQL) in the Machine Learning field has allowed facing Reinforcement Learning (RL) problems of unprecedented complexity. Unfortunately, complex DQL models usually find it difficult to converge to optimal policies due to the lack of exploration or sparse rewards. In order to overcome these drawbacks, pre-trained models are widely harnessed via Transfer Learning, extrapolating knowledge acquired in a source task to the target task. Besides, meta-heuristic optimization has been shown to reduce the lack of exploration of DQL models. This work proposes a MFO framework capable of simultaneously evolving several DQL models towards solving interrelated RL tasks. Specifically, our proposed framework blends together the benefits of meta-heuristic optimization, Transfer Learning and DQL to automate the process of knowledge transfer and policy learning of distributed RL agents. A thorough experimentation is presented and discussed so as to assess the performance of the framework, its comparison to the traditional methodology for Transfer Learning in terms of convergence, speed and policy quality , and the intertask relationships found and exploited over the search process.