Can We Optimize Deep RL Policy Weights as Trajectory Modeling?
This work addresses the challenge of scaling deep RL training by exploring policy weight trajectories, offering a novel approach that could impact RL efficiency, though it appears incremental in its application of Transformers to a new data type.
The paper tackles the problem of optimizing deep reinforcement learning policies by modeling the trajectory of policy network weights as a data modality, proposing Transformer as Implicit Policy Learner (TIPL) to fit implicit dynamics and optimize policies through inference.
Learning the optimal policy from a random network initialization is the theme of deep Reinforcement Learning (RL). As the scale of DRL training increases, treating DRL policy network weights as a new data modality and exploring the potential becomes appealing and possible. In this work, we focus on the policy learning path in deep RL, represented by the trajectory of network weights of historical policies, which reflects the evolvement of the policy learning process. Taking the idea of trajectory modeling with Transformer, we propose Transformer as Implicit Policy Learner (TIPL), which processes policy network weights in an autoregressive manner. We collect the policy learning path data by running independent RL training trials, with which we then train our TIPL model. In the experiments, we demonstrate that TIPL is able to fit the implicit dynamics of policy learning and perform the optimization of policy network by inference.