Conditional Frechet Inception Distance
This work provides a more accurate evaluation tool for conditional generative models, addressing limitations in existing metrics for researchers and practitioners in machine learning.
The paper tackled the problem of evaluating conditional generative models by developing a conditional version of the Frechet Inception Distance (CFID) metric, which outperforms classical FID and mean squared error in identifying specific failure modes, such as realistic but unrelated outputs or lack of diversity.
We consider distance functions between conditional distributions. We focus on the Wasserstein metric and its Gaussian case known as the Frechet Inception Distance (FID). We develop conditional versions of these metrics, analyze their relations and provide a closed form solution to the conditional FID (CFID) metric. We numerically compare the metrics in the context of performance evaluation of modern conditional generative models. Our results show the advantages of CFID compared to the classical FID and mean squared error (MSE) measures. In contrast to FID, CFID is useful in identifying failures where realistic outputs which are not related to their inputs are generated. On the other hand, compared to MSE, CFID is useful in identifying failures where a single realistic output is generated even though there is a diverse set of equally probable outputs.