Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale
This work addresses generalization challenges in deep reinforcement learning for researchers, showing incremental improvements through architectural modifications.
The paper tackled the problem of improving generalization in reinforcement learning on the ProcGen benchmark by implementing simple architectural changes like frame stacking, 3D convolutions, and scaling convolutional kernels, resulting in a 37.9% reduction in the optimality gap from 0.58 to 0.36.
We demonstrate that recent advances in reinforcement learning (RL) combined with simple architectural changes significantly improves generalization on the ProcGen benchmark. These changes are frame stacking, replacing 2D convolutional layers with 3D convolutional layers, and scaling up the number of convolutional kernels per layer. Experimental results using a single set of hyperparameters across all environments show a 37.9\% reduction in the optimality gap compared to the baseline (from 0.58 to 0.36). This performance matches or exceeds current state-of-the-art methods. The proposed changes are largely orthogonal and therefore complementary to the existing approaches for improving generalization in RL, and our results suggest that further exploration in this direction could yield substantial improvements in addressing generalization challenges in deep reinforcement learning.