An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
This work addresses the challenge of overfitting in RL for researchers, but it is incremental as it focuses on empirical analysis without introducing new methods.
The study tackled the problem of hyperparameter interdependence affecting reinforcement learning generalization, finding complex relationships and using empirical metrics like gradient cosine similarity to provide intuition.
Recent results in Reinforcement Learning (RL) have shown that agents with limited training environments are susceptible to a large amount of overfitting across many domains. A key challenge for RL generalization is to quantitatively explain the effects of changing parameters on testing performance. Such parameters include architecture, regularization, and RL-dependent variables such as discount factor and action stochasticity. We provide empirical results that show complex and interdependent relationships between hyperparameters and generalization. We further show that several empirical metrics such as gradient cosine similarity and trajectory-dependent metrics serve to provide intuition towards these results.