Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains
This addresses a bottleneck in VRAG systems for AI researchers and practitioners by improving visual reasoning capabilities, though it is incremental as it builds on existing VRAG and VLM methods.
The paper tackles the problem of visual information loss in Visual Retrieval-Augmented Generation (VRAG) frameworks by proposing Lang2Act, which uses self-emergent linguistic toolchains to enhance fine-grained visual perception and reasoning in Vision-Language Models (VLMs), resulting in performance improvements of over 4%.
Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.