Reinforcement Learning with Chromatic Networks for Compact Architecture Search
This work addresses the need for compact RL policies, particularly for mobile robotics with limited storage and computational resources, though it appears incremental as it builds on existing NAS and ES methods.
The paper tackled the problem of designing compact reinforcement learning policies by introducing a neural architecture search algorithm that combines ENAS and ES to efficiently search for edge-partitionings, resulting in policies with as few as 17 weight parameters, achieving over 90% compression compared to vanilla policies and 6x compression over state-of-the-art compact policies while maintaining good reward.
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way. By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings. For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as $17$ weight parameters, providing >90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward. We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.