\textit{Stochastic} MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
For RL practitioners, SMFP offers a practical policy class that balances expressivity and computational efficiency, improving performance over existing methods in continuous control benchmarks.
Stochastic MeanFlow Policies (SMFP) introduce a one-step generative policy class that combines tractable entropy estimation with multimodal expressivity, enabling stable off-policy mirror descent in RL. Across seven MuJoCo tasks, SMFP outperforms Gaussian and generative baselines while maintaining single-step inference.
Online off-policy reinforcement learning (RL) is shaped by two coupled choices: the policy class and the update rule. Gaussian policies are fast and have tractable entropy, but struggle with multimodal action distributions. Generative policies are more expressive, but often require iterative sampling or lack tractable entropy estimates. On the optimisation side, SAC-style soft policy improvement and mirror descent (MD) can be viewed as minimising different KL divergences: the former moves the policy towards a value-induced Boltzmann distribution, while the latter regularises each update against the previous policy. Combining entropy regularisation with an MD constraint is therefore attractive, as it supports exploration while stabilising policy improvement; however, the resulting target can be multimodal and is poorly matched by unimodal Gaussian policies. We propose Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that maps Gaussian noise to actions through a MeanFlow transformation. This stochastic reparameterisation yields a tractable entropy surrogate and allows MeanFlow policies to be trained within off-policy mirror descent under a unified objective for exploratory yet stable improvement. Across seven MuJoCo benchmarks, SMFP improves over Gaussian and generative baselines while retaining single-step inference efficiency.