Episodic Policy Gradient Training
This addresses the challenge of manual hyperparameter tuning for reinforcement learning practitioners, though it is an incremental improvement over existing methods.
The paper tackles the problem of hyperparameter tuning in reinforcement learning by introducing Episodic Policy Gradient Training (EPGT), which uses episodic memory to adaptively reconfigure hyperparameters on-the-fly, resulting in boosted performance for policy gradient algorithms in continuous and discrete environments.
We introduce a novel training procedure for policy gradient methods wherein episodic memory is used to optimize the hyperparameters of reinforcement learning algorithms on-the-fly. Unlike other hyperparameter searches, we formulate hyperparameter scheduling as a standard Markov Decision Process and use episodic memory to store the outcome of used hyperparameters and their training contexts. At any policy update step, the policy learner refers to the stored experiences, and adaptively reconfigures its learning algorithm with the new hyperparameters determined by the memory. This mechanism, dubbed as Episodic Policy Gradient Training (EPGT), enables an episodic learning process, and jointly learns the policy and the learning algorithm's hyperparameters within a single run. Experimental results on both continuous and discrete environments demonstrate the advantage of using the proposed method in boosting the performance of various policy gradient algorithms.