Critical analysis on the reproducibility of visual quality assessment using deep features
This work exposes reproducibility problems in visual quality assessment research, which is incremental as it critiques existing methods rather than introducing new ones.
The paper identifies complex data leakage issues in no-reference image and video quality assessment literature, showing that claimed high performance results are unattainable due to inappropriate use of test set information, with corrected performances dropping below state-of-the-art by a large margin.
Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the performances of the approaches drop even below the state-of-the-art by a large margin. Additionally, we investigate end-to-end variations to the discussed approaches, which do not improve upon the original.