Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
This work addresses a key implementation bottleneck in meta-reinforcement learning for researchers and practitioners, though it appears incremental as it builds upon existing approaches.
The paper tackles the challenge of estimating Hessian matrices for model-agnostic meta-reinforcement learning, which often leads to biased estimates, by introducing a unifying framework based on off-policy evaluation that unifies prior methods and enables new, easily implementable estimates with practical performance gains.
Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.