Assessing Policy, Loss and Planning Combinations in Reinforcement Learning using a New Modular Architecture
This work provides a modular tool for reinforcement learning researchers to study new environments and techniques, but it is incremental as it focuses on software architecture rather than fundamental algorithmic breakthroughs.
The authors tackled the complexity of model-based reinforcement learning agents by proposing a modular software architecture with reusable building blocks for planning algorithms, policies, and loss functions, and demonstrated its utility by testing combinations in environments like Cartpole, Minigrid, and Tictactoe, where a new averaged minimax planning algorithm achieved good results.
The model-based reinforcement learning paradigm, which uses planning algorithms and neural network models, has recently achieved unprecedented results in diverse applications, leading to what is now known as deep reinforcement learning. These agents are quite complex and involve multiple components, factors that can create challenges for research. In this work, we propose a new modular software architecture suited for these types of agents, and a set of building blocks that can be easily reused and assembled to construct new model-based reinforcement learning agents. These building blocks include planning algorithms, policies, and loss functions. We illustrate the use of this architecture by combining several of these building blocks to implement and test agents that are optimized to three different test environments: Cartpole, Minigrid, and Tictactoe. One particular planning algorithm, made available in our implementation and not previously used in reinforcement learning, which we called averaged minimax, achieved good results in the three tested environments. Experiments performed with this architecture have shown that the best combination of planning algorithm, policy, and loss function is heavily problem dependent. This result provides evidence that the proposed architecture, which is modular and reusable, is useful for reinforcement learning researchers who want to study new environments and techniques.