Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning
This work addresses the problem of understanding modularity-performance relationships for researchers in reinforcement learning and neural architecture design, but it is incremental as it builds on existing modularity studies without introducing a new method.
The paper investigates the unclear link between modularity and performance in neural networks for reinforcement learning, finding that optimal modularity is entangled with other network and environmental features, and direct optimization may not be beneficial, as shown through experiments on three tasks using neuroevolutionary algorithms.
Modularity has been widely studied as a mechanism to improve the capabilities of neural networks through various techniques such as hand-crafted modular architectures and automatic approaches. While these methods have sometimes shown improvements towards generalisation ability, robustness, and efficiency, the mechanisms that enable modularity to give performance advantages are unclear. In this paper, we investigate this issue and find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. Therefore, direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures and weights with varying levels of modularity. The structural modularity and performance of networks generated by the NeuroEvolution of Augmenting Topologies algorithm was assessed on three reinforcement learning tasks, with and without an additional modularity objective. The results of the quality-diversity optimisation algorithm, MAP-Elites, suggest intricate conditional relationships between modularity, performance, and other predefined network features.