Incrementally Learning Functions of the Return
This work addresses a limitation in reinforcement learning for practitioners needing more flexible value representations, though it appears incremental as it builds on existing TD methods.
The paper tackles the problem of estimating functions of the return in reinforcement learning, which standard temporal difference methods cannot handle directly, by proposing a method to learn the moments of the return online and using them in a Taylor expansion to approximate these functions.
Temporal difference methods enable efficient estimation of value functions in reinforcement learning in an incremental fashion, and are of broader interest because they correspond learning as observed in biological systems. Standard value functions correspond to the expected value of a sum of discounted returns. While this formulation is often sufficient for many purposes, it would often be useful to be able to represent functions of the return as well. Unfortunately, most such functions cannot be estimated directly using TD methods. We propose a means of estimating functions of the return using its moments, which can be learned online using a modified TD algorithm. The moments of the return are then used as part of a Taylor expansion to approximate analytic functions of the return.