Fully Variational Noise-Contrastive Estimation
This work addresses a theoretical bottleneck for researchers in machine learning, offering incremental improvements by extending NCE to latent variable models.
The paper tackles the challenge of applying noise-contrastive estimation (NCE) to latent variable models by developing a family of fully variational NCE methods based on proper scoring rules, resulting in tractable objectives that include variational autoencoders as a specific case.
By using the underlying theory of proper scoring rules, we design a family of noise-contrastive estimation (NCE) methods that are tractable for latent variable models. Both terms in the underlying NCE loss, the one using data samples and the one using noise samples, can be lower-bounded as in variational Bayes, therefore we call this family of losses fully variational noise-contrastive estimation. Variational autoencoders are a particular example in this family and therefore can be also understood as separating real data from synthetic samples using an appropriate classification loss. We further discuss other instances in this family of fully variational NCE objectives and indicate differences in their empirical behavior.