Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs
This addresses the problem of automating preference alignment for MLLMs to enhance trustworthiness, offering an incremental improvement over existing methods that rely on single evaluation models or closed-source evaluators.
The paper tackles the challenge of improving trustworthiness in Multimodal Large Language Models (MLLMs) by automating preference dataset construction, as human labeling is laborious. It proposes DecompGen, a decomposable framework using an ensemble of open-sourced expert models to break down responses into atomic verification tasks, resulting in MLLMs aligned with the constructed dataset showing improved trustworthiness.
Multimodal Large Language Models (MLLMs) can enhance trustworthiness by aligning with human preferences. As human preference labeling is laborious, recent works employ evaluation models for assessing MLLMs' responses, using the model-based assessments to automate preference dataset construction. This approach, however, faces challenges with MLLMs' lengthy and compositional responses, which often require diverse reasoning skills that a single evaluation model may not fully possess. Additionally, most existing methods rely on closed-source models as evaluators. To address limitations, we propose DecompGen, a decomposable framework that uses an ensemble of open-sourced expert models. DecompGen breaks down each response into atomic verification tasks, assigning each task to an appropriate expert model to generate fine-grained assessments. The DecompGen feedback is used to automatically construct our preference dataset, DGPref. MLLMs aligned with DGPref via preference learning show improvements in trustworthiness, demonstrating the effectiveness of DecompGen.