A Methodology to Evaluate Strategies Predicting Rankings on Unseen Domains
This addresses a practical challenge for researchers and practitioners in fields like computer vision who need to select optimal methods across diverse domains, though it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of predicting which methods will perform best in new domains without costly evaluations, by presenting a leave-one-domain-out methodology and demonstrating it with 30 strategies to rank 40 background subtraction methods on 53 video domains.
Frequently, multiple entities (methods, algorithms, procedures, solutions, etc.) can be developed for a common task and applied across various domains that differ in the distribution of scenarios encountered. For example, in computer vision, the input data provided to image analysis methods depend on the type of sensor used, its location, and the scene content. However, a crucial difficulty remains: can we predict which entities will perform best in a new domain based on assessments on known domains, without having to carry out new and costly evaluations? This paper presents an original methodology to address this question, in a leave-one-domain-out fashion, for various application-specific preferences. We illustrate its use with 30 strategies to predict the rankings of 40 entities (unsupervised background subtraction methods) on 53 domains (videos).