VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
This work addresses the problem of enhancing VLMs for agent applications like computer automation, offering a method to leverage low-quality datasets, though it is incremental as it builds on existing RL and SFT techniques.
The paper tackles the challenge of adapting vision-language models (VLMs) for interactive decision-making tasks where they lag behind LLMs, by proposing an offline-to-online reinforcement learning approach that fine-tunes VLMs to improve performance on agent tasks, demonstrating it with two VLMs across three multi-modal domains.
Recent research looks to harness the general knowledge and reasoning of large language models (LLMs) into agents that accomplish user-specified goals in interactive environments. Vision-language models (VLMs) extend LLMs to multi-modal data and provide agents with the visual reasoning necessary for new applications in areas such as computer automation. However, agent tasks emphasize skills where accessible open-weight VLMs lag behind their LLM equivalents. For example, VLMs are less capable of following an environment's strict output syntax requirements and are more focused on open-ended question answering. Overcoming these limitations requires supervised fine-tuning (SFT) on task-specific expert demonstrations. Our work approaches these challenges from an offline-to-online reinforcement learning (RL) perspective. RL lets us fine-tune VLMs to agent tasks while learning from the unsuccessful decisions of our own model or more capable (larger) models. We explore an off-policy RL solution that retains the stability and simplicity of the widely used SFT workflow while allowing our agent to self-improve and learn from low-quality datasets. We demonstrate this technique with two open-weight VLMs across three multi-modal agent domains.