FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
This work provides a more robust and category-agnostic approach to in-context object localization, which is crucial for applications like image editing and personalized visual search, especially for unnamed or instance-specific objects.
This paper addresses the challenge of in-context object localization (ICL) without category supervision, a critical limitation in existing vision-language models. The authors introduce a two-stage training framework that optimizes in-context attention and refines localization using Group Relative Policy Optimization (GRPO), resulting in a 7B-parameter model outperforming models up to 72B parameters.
In-context localization (ICL) seeks to localize a target object specified by a small set of support examples in a query image, operating on the fly without training or parameter updates. Despite rapid advances in vision-language models (VLMs), achieving category-agnostic and visually grounded ICL remains an open problem, even though it is essential for applications such as image editing, personalized visual search, and retrieval. Existing methods are fragile and rely on explicit category supervision, which not only limits applicability in realistic settings with unnamed or instance-specific objects but also introduces category bias that steers predictions toward semantic priors rather than visual evidence. We introduce a two-stage training framework that explicitly optimizes in-context attention between support bounding boxes and query images without category supervision. We further refine localization via reinforcement learning using Group Relative Policy Optimization (GRPO) to directly minimize localization error. This formulation enforces visual correspondence over semantic priors, yielding robust instance-level localization. Empirically, a 7B-parameter model trained with our objectives outperforms models up to 72B parameters, demonstrating that context-aware localization objectives can surpass scaling alone. Comprehensive ablations validate the contribution of each component.