Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation
This addresses a gap in multimodal AI by enabling generation to boost understanding, offering incremental but practical gains for researchers and developers working on unified models.
The paper tackles the problem of improving understanding in Unified Multimodal Models by proposing UniMRG, a post-training method that uses generation of multiple image representations (pixel, depth, segmentation) to enhance visual understanding, resulting in notable improvements in fine-grained perception, reduced hallucinations, and better spatial understanding.
Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.