Variational Information Bottleneck on Vector Quantized Autoencoders
This provides an information-theoretic interpretation for VQ-VAE, which is incremental as it explains an existing method without introducing new performance gains.
The paper tackles the problem of interpreting the Vector Quantized-Variational Autoencoder (VQ-VAE) by showing that its loss function can be derived from the variational deterministic information bottleneck principle, and that its EM-trained version approximates the variational information bottleneck principle.
In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expectation Maximization (EM) algorithm can be viewed as an approximation to the variational information bottleneck(VIB) principle.