Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning
This addresses a bottleneck in adaptive IRL for researchers and practitioners by providing a more efficient method for reconstructing loss functions from observed gradients.
The paper tackles the problem of inefficient gradient estimation in adaptive inverse reinforcement learning by using Malliavin calculus to estimate counterfactual gradients, achieving standard estimation rates instead of slow convergence from kernel smoothing.
Inverse reinforcement learning (IRL) recovers the loss function of a forward learner from its observed responses adaptive IRL aims to reconstruct the loss function of a forward learner by passively observing its gradients as it performs reinforcement learning (RL). This paper proposes a novel passive Langevin-based algorithm that achieves adaptive IRL. The key difficulty in adaptive IRL is that the required gradients in the passive algorithm are counterfactual, that is, they are conditioned on events of probability zero under the forward learner's trajectory. Therefore, naive Monte Carlo estimators are prohibitively inefficient, and kernel smoothing, though common, suffers from slow convergence. We overcome this by employing Malliavin calculus to efficiently estimate the required counterfactual gradients. We reformulate the counterfactual conditioning as a ratio of unconditioned expectations involving Malliavin quantities, thus recovering standard estimation rates. We derive the necessary Malliavin derivatives and their adjoint Skorohod integral formulations for a general Langevin structure, and provide a concrete algorithmic approach which exploits these for counterfactual gradient estimation.