UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation
This addresses the need for improved image perception and manipulation in practical applications, representing an incremental advance by building on insights from GPT-4o-Image.
The authors tackled the limitation of existing unified models in image perception and manipulation by proposing UniWorld-V1, a framework that uses semantic encoders for feature extraction, achieving impressive performance across diverse tasks with only 2.7M training data.
Although existing unified models achieve strong performance in vision-language understanding and text-to-image generation, they remain limited in addressing image perception and manipulation -- capabilities increasingly demanded in practical applications. Recently, OpenAI introduced the powerful GPT-4o-Image model, which showcases advanced capabilities in comprehensive image perception and manipulation, sparking widespread interest. Through carefully designed experiments, we observe that GPT-4o-Image likely relies on semantic encoders rather than VAEs for feature extraction, despite VAEs being commonly regarded as crucial for image manipulation tasks. Inspired by this insight, we propose UniWorld-V1, a unified generative framework built upon semantic features extracted from powerful multimodal large language models and contrastive semantic encoders. Using only 2.7M training data, UniWorld-V1 achieves impressive performance across diverse tasks, including image understanding, generation, manipulation, and perception. We fully open-source the UniWorld-V1 framework, including model weights, training and evaluation scripts, and datasets to promote reproducibility and further research.