Mixed Variational Flows for Discrete Variables
This work addresses a bottleneck in probabilistic modeling for discrete variables, offering a novel approach that could benefit practitioners in fields like natural language processing or genetics, though it appears incremental relative to existing flow-based methods.
The paper tackled the challenge of approximating discrete distributions with variational flows without using continuous embeddings, resulting in a method that provides more reliable approximations and is significantly faster to train.
Variational flows allow practitioners to learn complex continuous distributions, but approximating discrete distributions remains a challenge. Current methodologies typically embed the discrete target in a continuous space - usually via continuous relaxation or dequantization - and then apply a continuous flow. These approaches involve a surrogate target that may not capture the original discrete target, might have biased or unstable gradients, and can create a difficult optimization problem. In this work, we develop a variational flow family for discrete distributions without any continuous embedding. First, we develop a measure-preserving and discrete (MAD) invertible map that leaves the discrete target invariant, and then create a mixed variational flow (MAD Mix) based on that map. Our family provides access to i.i.d. sampling and density evaluation with virtually no tuning effort. We also develop an extension to MAD Mix that handles joint discrete and continuous models. Our experiments suggest that MAD Mix produces more reliable approximations than continuous-embedding flows while being significantly faster to train.