Poivre: Self-Refining Visual Pointing with Reinforcement Learning
This addresses a key limitation in vision-language models for tasks requiring precise object localization, offering a novel approach to enhance performance in visual pointing applications.
The paper tackles the problem of visual pointing in vision-language models, where current models lag behind human performance, by proposing a self-refining procedure called Poivre that uses reinforcement learning to iteratively refine coordinates, resulting in Poivre-7B achieving a new state of the art on Point-Bench with over 3% improvement over leading models.
Visual pointing, which aims to localize a target by predicting its coordinates on an image, has emerged as an important problem in the realm of vision-language models (VLMs). Despite its broad applicability, recent benchmarks show that current VLMs still fall far behind human performance on this task. A key limitation is that VLMs are typically required to complete the pointing task in a single step, akin to asking humans to point at an object without seeing their own fingers. To address this issue, we propose a simple yet effective self-refining procedure: Point, Visualize, then Refine (Poivre). This procedure enables a VLM to first mark its estimated point, then iteratively refine the coordinates if necessary. Inspired by advances of reasoning models in the natural language domain, we employ reinforcement learning (RL) to incentivize this self-refining ability. For the RL training, we design a neat process reward that is not only empirically effective but also grounded in appealing properties. Our trained model, Poivre-7B, sets a new state of the art on Point-Bench, outperforming both proprietary models such as Gemini-2.5-Pro and large open-source models such as Molmo-72B by over 3%. To support future research, we release our training and inference code, dataset, and the Poivre-7B checkpoint.