On The Statistical Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions
This work addresses theoretical gaps in understanding statistical properties of probability models used in discrete learning, with incremental contributions for researchers in generative and discriminative learning.
The paper investigates the statistical representation properties of Perturb-Softmax and Perturb-Argmax probability distributions, determining conditions under which they are complete and minimal, and extends this framework to models like Gaussian-Softmax and Gaussian-Argmax with experimental validation showing faster convergence rates.
The Gumbel-Softmax probability distribution allows learning discrete tokens in generative learning, while the Gumbel-Argmax probability distribution is useful in learning discrete structures in discriminative learning. Despite the efforts invested in optimizing these probability models, their statistical properties are under-explored. In this work, we investigate their representation properties and determine for which families of parameters these probability distributions are complete, i.e., can represent any probability distribution, and minimal, i.e., can represent a probability distribution uniquely. We rely on convexity and differentiability to determine these statistical conditions and extend this framework to general probability models, such as Gaussian-Softmax and Gaussian-Argmax. We experimentally validate the qualities of these extensions, which enjoy a faster convergence rate. We conclude the analysis by identifying two sets of parameters that satisfy these assumptions and thus admit a complete and minimal representation. Our contribution is theoretical with supporting practical evaluation.