Evaluating the Evaluators: Towards Human-aligned Metrics for Missing Markers Reconstruction
This addresses a domain-specific problem for animation researchers and practitioners by improving evaluation metrics for missing marker reconstruction, though it is incremental as it focuses on metric refinement rather than new reconstruction methods.
The paper tackles the problem of evaluating missing marker reconstruction in animation motion capture, showing that the standard mean square error metric does not correlate with subjective perception of fill quality, and introduces better-correlated metrics to drive progress.
Animation data is often obtained through optical motion capture systems, which utilize a multitude of cameras to establish the position of optical markers. However, system errors or occlusions can result in missing markers, the manual cleaning of which can be time-consuming. This has sparked interest in machine learning-based solutions for missing marker reconstruction in the academic community. Most academic papers utilize a simplistic mean square error as the main metric. In this paper, we show that this metric does not correlate with subjective perception of the fill quality. Additionally, we introduce and evaluate a set of better-correlated metrics that can drive progress in the field.