Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach
This work addresses computational bottlenecks in distributional RL for complex real-world decision-making and risk evaluation, representing an incremental improvement through a novel method for a known bottleneck.
The paper tackles the challenge of multi-dimensional distributional reinforcement learning in continuous state-action spaces by proposing a kernel mean embedding approach that replaces Wasserstein metrics with an integral probability metric, enabling efficient estimation. The results demonstrate robust off-policy evaluation and recovery of embeddings under mild assumptions like Lipschitz continuity and boundedness.
We propose an (offline) multi-dimensional distributional reinforcement learning framework (KE-DRL) that leverages Hilbert space mappings to estimate the kernel mean embedding of the multi-dimensional value distribution under a proposed target policy. In our setting, the state-action variables are multi-dimensional and continuous. By mapping probability measures into a reproducing kernel Hilbert space via kernel mean embeddings, our method replaces Wasserstein metrics with an integral probability metric. This enables efficient estimation in multi-dimensional state-action spaces and reward settings, where direct computation of Wasserstein distances is computationally challenging. Theoretically, we establish contraction properties of the distributional Bellman operator under our proposed metric involving the Matern family of kernels and provide uniform convergence guarantees. Simulations and empirical results demonstrate robust off-policy evaluation and recovery of the kernel mean embedding under mild assumptions, namely, Lipschitz continuity and boundedness of the kernels, highlighting the potential of embedding-based approaches in complex real-world decision-making scenarios and risk evaluation.