Backpropagation for Implicit Spectral Densities
This work addresses a bottleneck in unsupervised learning for high-dimensional data, offering a more general optimization approach, though it appears incremental as an extension of existing implicit likelihood methods.
The paper tackles the challenge of training unsupervised systems when explicit likelihoods are unavailable by introducing spectral backpropagation, a tool that optimizes implicit likelihoods without heavy assumptions, and applies it to GANs to reveal novel properties like aberrant high-likelihood outputs and quasi-disentangled factors in the generator.
Most successful machine intelligence systems rely on gradient-based learning, which is made possible by backpropagation. Some systems are designed to aid us in interpreting data when explicit goals cannot be provided. These unsupervised systems are commonly trained by backpropagating through a likelihood function. We introduce a tool that allows us to do this even when the likelihood is not explicitly set, by instead using the implicit likelihood of the model. Explicitly defining the likelihood often entails making heavy-handed assumptions that impede our ability to solve challenging tasks. On the other hand, the implicit likelihood of the model is accessible without the need for such assumptions. Our tool, which we call spectral backpropagation, allows us to optimize it in much greater generality than what has been attempted before. GANs can also be viewed as a technique for optimizing implicit likelihoods. We study them using spectral backpropagation in order to demonstrate robustness for high-dimensional problems, and identify two novel properties of the generator G: (1) there exist aberrant, nonsensical outputs to which G assigns very high likelihood, and (2) the eigenvectors of the metric induced by G over latent space correspond to quasi-disentangled explanatory factors.