CPPO: Contrastive Perception for Vision Language Policy Optimization
This work addresses the problem of efficient and scalable multimodal reasoning for AI systems, representing an incremental improvement over existing perception-rewarding approaches.
The paper tackles the challenge of finetuning vision-language models for multimodal reasoning by introducing CPPO, which uses entropy shifts to detect perception tokens and a contrastive loss to improve perception without extra models, achieving superior performance over prior methods.
We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision-language models (VLMs). While reinforcement learning (RL) has advanced reasoning in language models, extending it to multimodal reasoning requires improving both the perception and reasoning aspects. Prior works tackle this challenge mainly with explicit perception rewards, but disentangling perception tokens from reasoning tokens is difficult, requiring extra LLMs, ground-truth data, forced separation of perception from reasoning by policy model, or applying rewards indiscriminately to all output tokens. CPPO addresses this problem by detecting perception tokens via entropy shifts in the model outputs under perturbed input images. CPPO then extends the RL objective function with a Contrastive Perception Loss (CPL) that enforces consistency under information-preserving perturbations and sensitivity under information-removing ones. Experiments show that CPPO surpasses previous perception-rewarding methods, while avoiding extra models, making training more efficient and scalable.