Teaching a Robot to Walk Using Reinforcement Learning
This addresses the problem of adaptive locomotion for robots with complex dynamics, though it is incremental as it applies existing methods to a standard benchmark.
The paper tackled teaching a bipedal robot to walk using reinforcement learning, finding that augmented random search (ARS) successfully solved the OpenAI Gym BipedalWalker-v3 environment, while deep Q-learning performed poorly due to discretization issues.
Classical control techniques such as PID and LQR have been used effectively in maintaining a system state, but these techniques become more difficult to implement when the model dynamics increase in complexity and sensitivity. For adaptive robotic locomotion tasks with several degrees of freedom, this task becomes infeasible with classical control techniques. Instead, reinforcement learning can train optimal walking policies with ease. We apply deep Q-learning and augmented random search (ARS) to teach a simulated two-dimensional bipedal robot how to walk using the OpenAI Gym BipedalWalker-v3 environment. Deep Q-learning did not yield a high reward policy, often prematurely converging to suboptimal local maxima likely due to the coarsely discretized action space. ARS, however, resulted in a better trained robot, and produced an optimal policy which officially "solves" the BipedalWalker-v3 problem. Various naive policies, including a random policy, a manually encoded inch forward policy, and a stay still policy, were used as benchmarks to evaluate the proficiency of the learning algorithm results.