Summarizing the performances of a background subtraction algorithm measured on several videos
This work addresses a methodological issue for researchers comparing background subtraction algorithms, offering an incremental improvement over existing summarization methods.
The paper tackles the problem of inconsistent interpretation in summarizing background subtraction algorithm performances across multiple videos, presenting a theoretical approach that preserves relationships between performance indicators and providing formulas and an algorithm for calculation, demonstrated on the CDNET 2014 dataset.
There exist many background subtraction algorithms to detect motion in videos. To help comparing them, datasets with ground-truth data such as CDNET or LASIESTA have been proposed. These datasets organize videos in categories that represent typical challenges for background subtraction. The evaluation procedure promoted by their authors consists in measuring performance indicators for each video separately and to average them hierarchically, within a category first, then between categories, a procedure which we name "summarization". While the summarization by averaging performance indicators is a valuable effort to standardize the evaluation procedure, it has no theoretical justification and it breaks the intrinsic relationships between summarized indicators. This leads to interpretation inconsistencies. In this paper, we present a theoretical approach to summarize the performances for multiple videos that preserves the relationships between performance indicators. In addition, we give formulas and an algorithm to calculate summarized performances. Finally, we showcase our observations on CDNET 2014.