Barcode Method for Generative Model Evaluation driven by Topological Data Analysis
This work addresses the problem of more accurate and assumption-free evaluation for researchers and practitioners using generative models, though it is incremental as it builds on prior metrics.
The paper tackles the challenge of evaluating generative models in image synthesis by proposing the 'barcode' method, which outperforms existing metrics like Fréchet Inception Distance and precision-and-recall variants in assessing fidelity and diversity of GAN outputs, as demonstrated in experiments on real-world datasets.
Evaluating the performance of generative models in image synthesis is a challenging task. Although the Fréchet Inception Distance is a widely accepted evaluation metric, it integrates different aspects (e.g., fidelity and diversity) of synthesized images into a single score and assumes the normality of embedded vectors. Recent methods such as precision-and-recall and its variants such as density-and-coverage have been developed to separate fidelity and diversity based on k-nearest neighborhood methods. In this study, we propose an algorithm named barcode, which is inspired by the topological data analysis and is almost free of assumption and hyperparameter selections. In extensive experiments on real-world datasets as well as theoretical approach on high-dimensional normal samples, it was found that the 'usual' normality assumption of embedded vectors has several drawbacks. The experimental results demonstrate that barcode outperforms other methods in evaluating fidelity and diversity of GAN outputs. Official codes can be found in https://github.com/minjeekim00/Barcode.