Start Small, Think Big: Curriculum-based Relative Policy Optimization for Visual Grounding
This addresses performance issues in Visual Grounding for tasks with ambiguous textual descriptions, though it is an incremental method building on existing CoT and RL approaches.
The paper tackles the problem of performance degradation in reinforcement learning-based fine-tuned Chain-of-Thought reasoning for Visual Grounding tasks by proposing Curriculum-based Relative Policy Optimization (CuRPO), which structures training data from simpler to more challenging examples, resulting in improvements of up to +12.52 mAP on RefCOCO.
Chain-of-Thought (CoT) prompting has recently shown significant promise across various NLP and computer vision tasks by explicitly generating intermediate reasoning steps. However, we find that reinforcement learning (RL)-based fine-tuned CoT reasoning can paradoxically degrade performance in Visual Grounding tasks, particularly as CoT outputs become lengthy or complex. Additionally, our analysis reveals that increased dataset size does not always enhance performance due to varying data complexities. Motivated by these findings, we propose Curriculum-based Relative Policy Optimization (CuRPO), a novel training strategy that leverages CoT length and generalized Intersection over Union (gIoU) rewards as complexity indicators to progressively structure training data from simpler to more challenging examples. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and LISA datasets demonstrate the effectiveness of our approach. CuRPO consistently outperforms existing methods, including Visual-RFT, with notable improvements of up to +12.52 mAP on RefCOCO. Moreover, CuRPO exhibits exceptional efficiency and robustness, delivering strong localization performance even in few-shot learning scenarios, particularly benefiting tasks characterized by ambiguous and intricate textual descriptions.The code is released on https://github.com/qyoung-yan/CuRPO.