On the Evaluation of Video Keyframe Summaries using User Ground Truth
This work addresses the problem of standardizing evaluation for video summarization researchers, but it is incremental as it builds on existing ground truth and feature comparison methods.
The paper tackles the lack of formal evaluation methods for video keyframe summaries by proposing a discrimination capacity measure to quantify improvements over a uniform baseline using ground truth summaries, and finds that simple hue histograms suffice for comparison, with results based on the VSUMM collection involving 10 feature types and 6 matching methods.
Given the great interest in creating keyframe summaries from video, it is surprising how little has been done to formalise their evaluation and comparison. User studies are often carried out to demonstrate that a proposed method generates a more appealing summary than one or two rival methods. But larger comparison studies cannot feasibly use such user surveys. Here we propose a discrimination capacity measure as a formal way to quantify the improvement over the uniform baseline, assuming that one or more ground truth summaries are available. Using the VSUMM video collection, we examine 10 video feature types, including CNN and SURF, and 6 methods for matching frames from two summaries. Our results indicate that a simple frame representation through hue histograms suffices for the purposes of comparing keyframe summaries. We subsequently propose a formal protocol for comparing summaries when ground truth is available.