Adapting the Function Approximation Architecture in Online Reinforcement Learning
This work addresses the challenge of scalable reinforcement learning for problems where observation structure is not available, representing an incremental improvement over existing methods.
The paper tackled the problem of online reinforcement learning with unknown observation structures by proposing an adaptive architecture for value function approximation, which outperformed non-adaptive baselines and approached the performance of an architecture with side-channel information in a spatial domain with high-dimensional, stochastic observations.
The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide both optimization techniques and architectures for approximating nonlinear functions from noisy, high-dimensional observations. However, prevailing optimization techniques are not designed for strictly-incremental online updates. Nor are standard architectures designed for observations with an a priori unknown structure: for example, light sensors randomly dispersed in space. This paper proposes an online RL prediction algorithm with an adaptive architecture that efficiently finds useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. The algorithm outperforms non-adaptive baseline architectures and approaches the performance of an architecture given side-channel information. These results are a step towards scalable RL algorithms for more general problems, where the observation structure is not available.