Pioneering Perceptual Video Fluency Assessment: A Novel Task with Benchmark Dataset and Baseline
This work addresses the problem of accurately assessing video fluency for applications like streaming and gaming, but it is incremental as it builds on existing video quality assessment research by focusing on a specific sub-dimension.
The paper introduces Video Fluency Assessment (VFA) as a standalone task to estimate human perception of video fluency, addressing limitations in existing video quality assessment methods. It achieves state-of-the-art performance by proposing a benchmark dataset, FluVid, and a baseline model, FluNet, with temporal permuted self-attention.
Accurately estimating humans' subjective feedback on video fluency, e.g., motion consistency and frame continuity, is crucial for various applications like streaming and gaming. Yet, it has long been overlooked, as prior arts have focused on solving it in the video quality assessment (VQA) task, merely as a sub-dimension of overall quality. In this work, we conduct pilot experiments and reveal that current VQA predictions largely underrepresent fluency, thereby limiting their applicability. To this end, we pioneer Video Fluency Assessment (VFA) as a standalone perceptual task focused on the temporal dimension. To advance VFA research, 1) we construct a fluency-oriented dataset, FluVid, comprising 4,606 in-the-wild videos with balanced fluency distribution, featuring the first-ever scoring criteria and human study for VFA. 2) We develop a large-scale benchmark of 23 methods, the most comprehensive one thus far on FluVid, gathering insights for VFA-tailored model designs. 3) We propose a baseline model called FluNet, which deploys temporal permuted self-attention (T-PSA) to enrich input fluency information and enhance long-range inter-frame interactions. Our work not only achieves state-of-the-art performance but, more importantly, offers the community a roadmap to explore solutions for VFA.