Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits
This addresses the challenge of sample-inefficient and unstable RL fine-tuning for EDPs in robotics or control tasks, though it is incremental as it builds on existing EDP and RL methods.
The paper tackled the problem of fine-tuning equivariant diffusion policies (EDPs) with reinforcement learning (RL) by developing a symmetry-aware steering framework, showing that it enhances sample efficiency, prevents value divergence, and achieves strong policy improvements even with limited demonstrations.
Equivariant diffusion policies (EDPs) combine the generative expressivity of diffusion models with the strong generalization and sample efficiency afforded by geometric symmetries. While steering these policies with reinforcement learning (RL) offers a promising mechanism for fine-tuning beyond demonstration data, directly applying standard (non-equivariant) RL can be sample-inefficient and unstable, as it ignores the symmetries that EDPs are designed to exploit. In this paper, we theoretically establish that the diffusion process of an EDP is equivariant, which in turn induces a group-invariant latent-noise MDP that is well-suited for equivariant diffusion steering. Building on this theory, we introduce a principled symmetry-aware steering framework and compare standard, equivariant, and approximately equivariant RL strategies through comprehensive experiments across tasks with varying degrees of symmetry. While we identify the practical boundaries of strict equivariance under symmetry breaking, we show that exploiting symmetry during the steering process yields substantial benefits-enhancing sample efficiency, preventing value divergence, and achieving strong policy improvements even when EDPs are trained from extremely limited demonstrations.