A Reference Architecture of Reinforcement Learning Frameworks
This paper addresses the lack of a common reference architecture for reinforcement learning frameworks, which is a problem for researchers and developers working with diverse RL technologies.
The authors analyzed 18 state-of-the-practice reinforcement learning (RL) frameworks to identify recurring architectural components and their relationships. They codified these findings into a reference architecture (RA) for RL frameworks, which can be used for comparison, evaluation, and integration.
The surge in reinforcement learning (RL) applications gave rise to diverse supporting technology, such as RL frameworks. However, the architectural patterns of these frameworks are inconsistent across implementations and there exists no reference architecture (RA) to form a common basis of comparison, evaluation, and integration. To address this gap, we propose an RA of RL frameworks. Through a grounded theory approach, we analyze 18 state-of-the-practice RL frameworks and, by that, we identify recurring architectural components and their relationships, and codify them in an RA. To demonstrate our RA, we reconstruct characteristic RL patterns. Finally, we identify architectural trends, e.g., commonly used components, and outline paths to improving RL frameworks.