Can We Trust Deep Speech Prior?
This work highlights a cautionary issue for researchers and practitioners in speech enhancement, as it identifies a fundamental limitation in applying deep priors, making it incremental by building on existing deep prior methods.
The paper investigates the reliability of deep speech priors for speech enhancement, finding that while they can achieve reasonable performance, the results may be suboptimal due to a mismatch between deep generative model flexibility and maximum-likelihood training.
Recently, speech enhancement (SE) based on deep speech prior has attracted much attention, such as the variational auto-encoder with non-negative matrix factorization (VAE-NMF) architecture. Compared to conventional approaches that represent clean speech by shallow models such as Gaussians with a low-rank covariance, the new approach employs deep generative models to represent the clean speech, which often provides a better prior. Despite the clear advantage in theory, we argue that deep priors must be used with much caution, since the likelihood produced by a deep generative model does not always coincide with the speech quality. We designed a comprehensive study on this issue and demonstrated that based on deep speech priors, a reasonable SE performance can be achieved, but the results might be suboptimal. A careful analysis showed that this problem is deeply rooted in the disharmony between the flexibility of deep generative models and the nature of the maximum-likelihood (ML) training.