VTAMIQ: Transformers for Attention Modulated Image Quality Assessment
This addresses the need for more accurate and generalizable IQA methods for image analysis, though it is incremental as it builds on existing transformer and attention techniques.
The authors tackled the problem of full-reference Image Quality Assessment (IQA) by proposing VTAMIQ, a transformer-based method that uses attention mechanisms to model global interdependencies between patches, achieving competitive or state-of-the-art performance on existing datasets and significantly outperforming previous metrics in cross-database evaluations.
Following the major successes of self-attention and Transformers for image analysis, we investigate the use of such attention mechanisms in the context of Image Quality Assessment (IQA) and propose a novel full-reference IQA method, Vision Transformer for Attention Modulated Image Quality (VTAMIQ). Our method achieves competitive or state-of-the-art performance on the existing IQA datasets and significantly outperforms previous metrics in cross-database evaluations. Most patch-wise IQA methods treat each patch independently; this partially discards global information and limits the ability to model long-distance interactions. We avoid this problem altogether by employing a transformer to encode a sequence of patches as a single global representation, which by design considers interdependencies between patches. We rely on various attention mechanisms -- first with self-attention within the Transformer, and second with channel attention within our difference modulation network -- specifically to reveal and enhance the more salient features throughout our architecture. With large-scale pre-training for both classification and IQA tasks, VTAMIQ generalizes well to unseen sets of images and distortions, further demonstrating the strength of transformer-based networks for vision modelling.