Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learning
This addresses the challenge of enabling vision language models to perform better in complex decision-making tasks with reduced reliance on scarce paired image-text data, representing an incremental improvement in method.
The paper tackles the problem of vision language models lacking sophisticated reasoning for complex decision-making by showing that replacing visual scenes with textual descriptions enables strong decision-making performance, and proposes Praxis-VLM, which uses text-driven reinforcement learning to instill reasoning skills that transfer to multimodal inference, outperforming standard supervised fine-tuning across benchmarks.
Vision Language Models exhibit impressive performance for various tasks, yet they often lack the sophisticated situational reasoning required for complex decision-making. This paper shows that VLMs can achieve surprisingly strong decision-making performance when visual scenes are replaced by textual descriptions, suggesting foundational reasoning can be effectively learned from language. Motivated by this insight, we propose Praxis-VLM, a reasoning VLM for vision-grounded decision-making. Praxis-VLM employs the GRPO algorithm on textual scenarios to instill robust reasoning capabilities, where models learn to evaluate actions and their consequences. These reasoning skills, acquired purely from text, successfully transfer to multimodal inference with visual inputs, significantly reducing reliance on scarce paired image-text training data. Experiments across diverse decision-making benchmarks demonstrate that Praxis-VLM substantially outperforms standard supervised fine-tuning, exhibiting superior performance and generalizability. Further analysis confirms that our models engage in explicit and effective reasoning, underpinning their enhanced performance and adaptability.