Evaluating generative audio systems and their metrics
This addresses the challenge of quantifying and comparing audio synthesis systems for researchers and practitioners, but it is incremental as it critiques existing methods without proposing a new solution.
The paper tackled the problem of inconsistent and perceptually irrelevant metrics for evaluating generative audio systems, finding that current objective metrics are insufficient to describe perceptual quality.
Recent years have seen considerable advances in audio synthesis with deep generative models. However, the state-of-the-art is very difficult to quantify; different studies often use different evaluation methodologies and different metrics when reporting results, making a direct comparison to other systems difficult if not impossible. Furthermore, the perceptual relevance and meaning of the reported metrics in most cases unknown, prohibiting any conclusive insights with respect to practical usability and audio quality. This paper presents a study that investigates state-of-the-art approaches side-by-side with (i) a set of previously proposed objective metrics for audio reconstruction, and with (ii) a listening study. The results indicate that currently used objective metrics are insufficient to describe the perceptual quality of current systems.