GIPO: Gaussian Importance Sampling Policy Optimization
This work addresses data efficiency issues in reinforcement learning for multimodal agents, offering incremental improvements over existing clipping-based methods.
The paper tackled the problem of poor data efficiency in reinforcement learning for multimodal agents by proposing GIPO, a policy optimization method using Gaussian importance sampling, which achieved state-of-the-art performance across various replay buffer sizes with improved sample efficiency and stability.
Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.