Path-Coupled Bellman Flows for Distributional Reinforcement Learning
For researchers in distributional reinforcement learning, PCBF offers a principled flow-based approach that addresses key limitations of prior methods, though improvements are incremental over existing flow-based DRL.
The paper introduces Path-Coupled Bellman Flows (PCBF), a continuous-time distributional RL method that uses source-consistent Bellman-coupled paths to avoid boundary mismatch and high-variance bootstrapping. Experiments on MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, with competitive offline RL performance.
Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundary mismatch} at the flow source or from \emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using \textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $λ$-parameterized control-variate target: $λ{=}0$ recovers an unbiased sample Bellman target, while $λ{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.