Policy Gradient RL Algorithms as Directed Acyclic Graphs
This work is an incremental extension of an existing framework for researchers working on automated RL algorithm design.
This paper extends a framework that represents RL algorithms as Directed Acyclic Graphs (DAGs) to include Policy Gradient algorithms. The authors propose DAG representations for five specific Policy Gradient algorithms: VPG, PPO, DDPG, TD3, and SAC.
Meta Reinforcement Learning (RL) methods focus on automating the design of RL algorithms that generalize to a wide range of environments. The framework introduced in (Anonymous, 2020) addresses the problem by representing different RL algorithms as Directed Acyclic Graphs (DAGs), and using an evolutionary meta learner to modify these graphs and find good agent update rules. While the search language used to generate graphs in the paper serves to represent numerous already-existing RL algorithms (e.g., DQN, DDQN), it has limitations when it comes to representing Policy Gradient algorithms. In this work we try to close this gap by extending the original search language and proposing graphs for five different Policy Gradient algorithms: VPG, PPO, DDPG, TD3, and SAC.