FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning
This addresses a key bottleneck in RL for improving convergence and performance, though it appears incremental as it builds on existing flow matching techniques.
The paper tackles the problem of reliable value estimation in reinforcement learning by proposing FlowCritic, a generative paradigm that models value distributions using flow matching, resulting in improved expressiveness for complex distributions.
Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.