Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
This work provides an incremental improvement for researchers and practitioners working with generative models, offering a new method to compare probability distributions that enhances training stability and performance.
This paper introduces Conditional Transport (CT) to compare probability distributions, which is constructed using the chain rule and Bayes' theorem. When integrated into generative adversarial networks, CT consistently improves performance on various benchmark datasets by balancing mode-covering and mode-seeking behaviors and resisting mode collapse.
To measure the difference between two probability distributions, referred to as the source and target, respectively, we exploit both the chain rule and Bayes' theorem to construct conditional transport (CT), which is constituted by both a forward component and a backward one. The forward CT is the expected cost of moving a source data point to a target one, with their joint distribution defined by the product of the source probability density function (PDF) and a source-dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost can be approximated by replacing the source and target PDFs with their discrete empirical distributions supported on mini-batches, making it amenable to implicit distributions and stochastic gradient descent-based optimization. When applied to train a generative model, CT is shown to strike a good balance between mode-covering and mode-seeking behaviors and strongly resist mode collapse. On a wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing generative adversarial network with CT is shown to consistently improve the performance. PyTorch code is provided.