Technical Report on Visual Quality Assessment for Frame Interpolation
This work addresses the gap in evaluating optical flow and frame interpolation algorithms for real-world applications, though it is incremental as it applies existing subjective assessment methods to an existing benchmark.
The authors tackled the problem that objective metrics like mean square error do not fully capture user experience in frame interpolation, by conducting a crowdsourced subjective quality assessment study on the Middlebury benchmark. They re-ranked 141 algorithms based on visual quality, showing the need for this as an additional evaluation metric.
Current benchmarks for optical flow algorithms evaluate the estimation quality by comparing their predicted flow field with the ground truth, and additionally may compare interpolated frames, based on these predictions, with the correct frames from the actual image sequences. For the latter comparisons, objective measures such as mean square errors are applied. However, for applications like image interpolation, the expected user's quality of experience cannot be fully deduced from such simple quality measures. Therefore, we conducted a subjective quality assessment study by crowdsourcing for the interpolated images provided in one of the optical flow benchmarks, the Middlebury benchmark. We used paired comparisons with forced choice and reconstructed absolute quality scale values according to Thurstone's model using the classical least squares method. The results give rise to a re-ranking of 141 participating algorithms w.r.t. visual quality of interpolated frames mostly based on optical flow estimation. Our re-ranking result shows the necessity of visual quality assessment as another evaluation metric for optical flow and frame interpolation benchmarks.