Recover Subjective Quality Scores from Noisy Measurements
This work addresses the challenge of improving video quality assessment for applications like streaming and broadcasting by enhancing the accuracy of metrics trained on subjective data, though it is incremental in nature.
The paper tackles the problem of recovering subjective quality scores from noisy measurements by jointly estimating video quality, subject bias/consistency, and content ambiguity using maximum likelihood estimation, resulting in tighter confidence intervals and better outlier handling without requiring z-scoring or subject rejection.
Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.