Kernel Exponential Family Estimation via Doubly Dual Embedding
This work addresses a fundamental challenge in kernel-based density estimation for machine learning, offering a more efficient and statistically robust method.
The paper tackles the problem of estimating exponential family distributions with parameters in a reproducing kernel Hilbert space by introducing a doubly dual embedding technique that avoids partition function computation, resulting in improved memory and time efficiency and stronger statistical properties compared to score matching estimators.
We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space. Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the development of a flexible sampling strategy that amortizes the cost of Monte-Carlo sampling in the inference stage. The resulting estimator can be easily generalized to kernel conditional exponential families. We establish a connection between kernel exponential family estimation and MMD-GANs, revealing a new perspective for understanding GANs. Compared to the score matching based estimators, the proposed method improves both memory and time efficiency while enjoying stronger statistical properties, such as fully capturing smoothness in its statistical convergence rate while the score matching estimator appears to saturate. Finally, we show that the proposed estimator empirically outperforms state-of-the-art