TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models
This addresses the need for more reliable evaluation in generative modeling, though it is incremental as it builds on existing precision and recall variants.
The paper tackles the problem of unreliable evaluation metrics for generative models by proposing TopP&R, a robust metric that uses topological and statistical methods for support estimation, showing it is resilient to noise and accurately captures sample trends.
We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for rigorous support estimation. Existing metrics, such as Inception Score (IS), Frechet Inception Distance (FID), and the variants of Precision and Recall (P&R), heavily rely on supports that are estimated from sample features. However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it. In this paper, we propose Topological Precision and Recall (TopP&R, pronounced 'topper'), which provides a systematic approach to estimating supports, retaining only topologically and statistically important features with a certain level of confidence. This not only makes TopP&R strong for noisy features, but also provides statistical consistency. Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples. To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support and provides its statistical consistency under noise.