Scalable Expectation Estimation with Subtractive Mixture Models
This work addresses a scalability problem for researchers and practitioners using SMMs in high-dimensional expectation estimation, though it appears incremental as it builds on existing SMM frameworks.
The paper tackles the challenge of sampling from subtractive mixture models (SMMs) with negative coefficients, which is computationally expensive in high dimensions, by introducing an unbiased importance sampling estimator (ΔEx) that avoids direct sampling. The result shows that ΔEx achieves comparable estimation quality to auto-regressive methods while being significantly faster in Monte Carlo estimation.
Many Monte Carlo (MC) and importance sampling (IS) methods use mixture models (MMs) for their simplicity and ability to capture multimodal distributions. Recently, subtractive mixture models (SMMs), i.e. MMs with negative coefficients, have shown greater expressiveness and success in generative modeling. However, their negative parameters complicate sampling, requiring costly auto-regressive techniques or accept-reject algorithms that do not scale in high dimensions. In this work, we use the difference representation of SMMs to construct an unbiased IS estimator ($Δ\text{Ex}$) that removes the need to sample from the SMM, enabling high-dimensional expectation estimation with SMMs. In our experiments, we show that $Δ\text{Ex}$ can achieve comparable estimation quality to auto-regressive sampling while being considerably faster in MC estimation. Moreover, we conduct initial experiments with $Δ\text{Ex}$ using hand-crafted proposals, gaining first insights into how to construct safe proposals for $Δ\text{Ex}$.