DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
This work addresses the problem of improving training efficiency and performance for multimodal PRMs, which is crucial for advancing AI reasoning tasks, though it appears incremental as it builds on prior domain-level reweighting methods.
The paper tackles the challenges of training multimodal process reward models (PRMs) due to distribution shift and quality imbalance by introducing DreamPRM-1.5, an instance-level reweighting framework that assigns adaptive weights to training examples via bi-level optimization, achieving 84.6 accuracy on MMMU validation and 31.3 accuracy on R-Bench-V, with first-place results on public multimodal reasoning leaderboards when paired with a leading backbone.
Training multimodal process reward models (PRMs) is hard due to (i) distribution shift between training set and test set and (ii) quality imbalance across training data samples. While domain-level reweighting (e.g., DreamPRM) aligns training with test-time objectives, it leaves a clear gap to an oracle upper bound (pass@N), even under a "sanity check" that uses test set data to probe headroom -- pointing to meta-level under-parameterization. We introduce DreamPRM-1.5, an instance-level reweighting framework that assigns an adaptive weight to every training example via bi-level optimization. To realize instance reweighting across scales, we develop two complementary regimes: Instance Table, which learns explicit per-sample weights and excels on small/medium data, and Instance Net, a lightweight neural network that generalizes better and scales to large corpora. A practical, stable training recipe -- time-scale matching between upper/lower updates, cold-start initialization, and bounded-range weights -- prevents divergence. Integrated with test-time scaling, DreamPRM-1.5 attains 84.6 accuracy on the MMMU validation set, 31.3 accuracy on R-Bench-V and, when paired with a leading backbone (e.g., GPT-5-mini), achieves first-place results on public multimodal reasoning leaderboards. Moreover, extensive experiments, including benchmark evaluations, baseline comparisons, and a sanity check, demonstrate that DreamPRM-1.5 closes the gap toward the oracle, achieves leading performance, and trains stably.