VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment
This work addresses the need for versatile and transferable models in visual quality assessment, though it appears incremental by building on existing LMM approaches.
The authors tackled the problem of limited generalization and transferability in visual quality assessment (VQualA) large multi-modal models (LMMs) by proposing a vision-encoder-centered pre-training pipeline, resulting in a model zoo with strong zero-shot performance and efficient training requiring less than 1/1000 of pre-training data for comparable performance.
Developing a robust visual quality assessment (VQualA) large multi-modal model (LMM) requires achieving versatility, powerfulness, and transferability. However, existing VQualA LMMs typically focus on a single task and rely on full-parameter fine-tuning, which makes them prone to overfitting on specific modalities or task types, thereby limiting their generalization capacity and transferability. To address this, we propose a vision-encoder-centered generative pre-training pipeline and develop the VITAL-Series LMMs. (1) We adopt a machine-executed annotation-scrutiny paradigm, constructing over 4.5M vision-language (VL) pairs-the largest VQualA training dataset to date. (2) We employ a multi-task training workflow that simultaneously enhances the model's quantitative scoring precision and strengthens its capability for quality interpretation across both image and video modalities. (3) Building upon the vision encoder, we realize an efficient model zoo extension: the model zoo exhibits strong zero-shot performance, and each paired decoder requires only a swift warm-up using less than 1/1000 of the pre-training data to achieve performance comparable to the fully trained counterpart. Overall, our work lays a cornerstone for advancing toward the foundation LMM for VQualA.