STORM: Benchmarking Visual Rating of MLLMs with a Comprehensive Ordinal Regression Dataset
This work addresses a domain-specific problem for researchers and developers in AI visual rating, such as image quality assessment and medical imaging, by providing a new benchmark and method, though it is incremental as it builds on existing ordinal regression tasks.
The authors tackled the problem of multi-modal large language models (MLLMs) underperforming in visual rating tasks due to a lack of datasets and benchmarks, by collecting STORM, a comprehensive dataset with 655K image-level pairs across 14 datasets, and proposing a coarse-to-fine processing pipeline that improves performance.
Visual rating is an essential capability of artificial intelligence (AI) for multi-dimensional quantification of visual content, primarily applied in ordinal regression (OR) tasks such as image quality assessment, facial age estimation, and medical image grading. However, current multi-modal large language models (MLLMs) under-perform in such visual rating ability while also suffering the lack of relevant datasets and benchmarks. In this work, we collect and present STORM, a data collection and benchmark for Stimulating Trustworthy Ordinal Regression Ability of MLLMs for universal visual rating. STORM encompasses 14 ordinal regression datasets across five common visual rating domains, comprising 655K image-level pairs and the corresponding carefully curated VQAs. Importantly, we also propose a coarse-to-fine processing pipeline that dynamically considers label candidates and provides interpretable thoughts, providing MLLMs with a general and trustworthy ordinal thinking paradigm. This benchmark aims to evaluate the all-in-one and zero-shot performance of MLLMs in scenarios requiring understanding of the essential common ordinal relationships of rating labels. Extensive experiments demonstrate the effectiveness of our framework and shed light on better fine-tuning strategies. The STORM dataset, benchmark, and pre-trained models are available on the following webpage to support further research in this area. Datasets and codes are released on the project page: https://storm-bench.github.io/.