UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark
This provides a more comprehensive evaluation tool for researchers and developers working on multimodal AI, though it is incremental as it builds on existing benchmark practices.
The authors tackled the lack of integrated evaluation for unified multimodal models (UMMs) by proposing UmniBench, a benchmark that assesses understanding, generation, and editing abilities in a single process, covering 13 domains and over 200 concepts, and benchmarking 24 models.
Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model.