A New Perspective on Precision and Recall for Generative Models
This work addresses the evaluation problem for generative models in AI, offering a more nuanced analysis than scalar metrics, though it is incremental as it builds on existing Precision and Recall concepts.
The paper tackles the challenge of estimating Precision and Recall curves for generative models by proposing a new framework based on binary classification, resulting in a thorough statistical analysis and a minimax upper bound on the estimation risk.
With the recent success of generative models in image and text, the question of their evaluation has recently gained a lot of attention. While most methods from the state of the art rely on scalar metrics, the introduction of Precision and Recall (PR) for generative model has opened up a new avenue of research. The associated PR curve allows for a richer analysis, but their estimation poses several challenges. In this paper, we present a new framework for estimating entire PR curves based on a binary classification standpoint. We conduct a thorough statistical analysis of the proposed estimates. As a byproduct, we obtain a minimax upper bound on the PR estimation risk. We also show that our framework extends several landmark PR metrics of the literature which by design are restrained to the extreme values of the curve. Finally, we study the different behaviors of the curves obtained experimentally in various settings.