Improving Generalization in Mountain Car Through the Partitioned Parameterized Policy Approach via Quasi-Stochastic Gradient Descent
This addresses the problem of poor generalization and trapping in circular trajectories for reinforcement learning in the Mountain Car environment, representing an incremental improvement over prior methods.
The paper tackled the problem of finding a control policy to minimize time in the Mountain Car environment by optimizing parameterized nonlinear feedback policies using quasi-Stochastic Gradient Descent (qSGD). The result was that a partitioned parameterized policy approach, which learns optimal parameters for different state regions, outperformed uniform approaches and improved generalization, reducing issues like circular trajectories.
The reinforcement learning problem of finding a control policy that minimizes the minimum time objective for the Mountain Car environment is considered. Particularly, a class of parameterized nonlinear feedback policies is optimized over to reach the top of the highest mountain peak in minimum time. The optimization is carried out using quasi-Stochastic Gradient Descent (qSGD) methods. In attempting to find the optimal minimum time policy, a new parameterized policy approach is considered that seeks to learn an optimal policy parameter for different regions of the state space, rather than rely on a single macroscopic policy parameter for the entire state space. This partitioned parameterized policy approach is shown to outperform the uniform parameterized policy approach and lead to greater generalization than prior methods, where the Mountain Car became trapped in circular trajectories in the state space.