Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
This addresses computational bottlenecks for researchers and practitioners using generative models in reinforcement learning, though it is an incremental improvement over existing diffusion-based approaches.
The authors tackled the computational inefficiency of diffusion models in reinforcement learning by proposing consistency models as policy representations, achieving comparable performance to diffusion policies with significant speed improvements (up to 10x faster sampling).
Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.