Learning RoboCup-Keepaway with Kernels
This work addresses the challenge of real-time, cooperative learning in complex environments for robotics and AI simulations, representing an incremental improvement over prior methods.
The paper tackled the high-dimensional and stochastic reinforcement learning problem of 3vs2 keepaway in RoboCup simulated soccer by applying kernel-based methods with approximate policy iteration, resulting in learned behavior that clearly outperformed earlier tilecoding-based results.
We apply kernel-based methods to solve the difficult reinforcement learning problem of 3vs2 keepaway in RoboCup simulated soccer. Key challenges in keepaway are the high-dimensionality of the state space (rendering conventional discretization-based function approximation like tilecoding infeasible), the stochasticity due to noise and multiple learning agents needing to cooperate (meaning that the exact dynamics of the environment are unknown) and real-time learning (meaning that an efficient online implementation is required). We employ the general framework of approximate policy iteration with least-squares-based policy evaluation. As underlying function approximator we consider the family of regularization networks with subset of regressors approximation. The core of our proposed solution is an efficient recursive implementation with automatic supervised selection of relevant basis functions. Simulation results indicate that the behavior learned through our approach clearly outperforms the best results obtained earlier with tilecoding by Stone et al. (2005).