SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
This addresses a key bottleneck in multimodal AI for applications requiring accurate cross-modal tasks, though it is an incremental advancement building on existing LMMs.
The paper tackles the problem of inaccurate vision-language alignment in large multimodal models (LMMs), which often generate text responses contradicting visual inputs or fail to follow text-to-image prompts, by proposing SUDER, a self-supervised framework that uses dual self-rewards to reinforce understanding and generation capabilities, achieving remarkable improvements in text-to-image tasks without external supervision.
Building upon large language models (LLMs), recent large multimodal models (LMMs) unify cross-model understanding and generation into a single framework. However, LMMs still struggle to achieve accurate vision-language alignment, prone to generating text responses contradicting the visual input or failing to follow the text-to-image prompts. Current solutions require external supervision (e.g., human feedback or reward models) and only address unidirectional tasks-either understanding or generation. In this work, based on the observation that understanding and generation are naturally inverse dual tasks, we propose \textbf{SUDER} (\textbf{S}elf-improving \textbf{U}nified LMMs with \textbf{D}ual s\textbf{E}lf-\textbf{R}ewards), a framework reinforcing the understanding and generation capabilities of LMMs with a self-supervised dual reward mechanism. SUDER leverages the inherent duality between understanding and generation tasks to provide self-supervised optimization signals for each other. Specifically, we sample multiple outputs for a given input in one task domain, then reverse the input-output pairs to compute the dual likelihood within the model as self-rewards for optimization. Extensive experimental results on visual understanding and generation benchmarks demonstrate that our method can effectively enhance the performance of the model without any external supervision, especially achieving remarkable improvements in text-to-image tasks.