Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program
This addresses the problem of inefficient reward modeling in multimodal AI for researchers and practitioners, though it appears incremental as it builds on existing CoT and reward model techniques.
The paper tackles challenges in applying reward signals to multimodal tasks by proposing SVIP, a method to automatically train a step-level Chain-of-Thought reward model, which improves multimodal large language model performance by reducing hallucinations and enhancing reasoning on benchmarks.
Recent advancements in reward signal usage for Large Language Models (LLMs) are remarkable. However, significant challenges exist when transitioning reward signal to the multimodal domain, including labor-intensive annotations, over-reliance on one-step rewards, and inadequate evaluation. To address these issues, we propose SVIP, a novel approach to train a step-level multi-dimensional Chain-of-Thought~(CoT) reward model automatically. It generates code for solving visual tasks and transforms the analysis of code blocks into the evaluation of CoT step as training samples. Then, we train SVIP-Reward model using a multi-head attention mechanism called TriAtt-CoT. The advantages of SVIP-Reward are evident throughout the entire process of MLLM. We also introduce a benchmark for CoT reward model training and testing. Experimental results demonstrate that SVIP-Reward improves MLLM performance across training and inference-time scaling, yielding better results on benchmarks while reducing hallucinations and enhancing reasoning ability.