The Role of ImageNet Classes in Fréchet Inception Distance
This work highlights a critical flaw in FID for evaluating generative models, which is important for researchers and practitioners relying on this metric, though it is incremental in identifying a known issue.
The paper investigates why Fréchet Inception Distance (FID) sometimes disagrees with human judgment in generative modeling, showing that aligning histograms of ImageNet classifications can reduce FID without improving image quality, as demonstrated by FastGAN achieving a comparable FID to StyleGAN2 despite worse human evaluation.
Fréchet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially -- without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.