Ratio Divergence Learning Using Target Energy in Restricted Boltzmann Machines: Beyond Kullback--Leibler Divergence Learning
This addresses a key bottleneck in training discrete energy-based models for machine learning applications, offering a more stable and effective method, though it appears incremental as it builds on existing divergence concepts.
The authors tackled the problem of underfitting and mode-collapse in discrete energy-based models like restricted Boltzmann machines by proposing ratio divergence learning, which uses a target energy function to combine forward and reverse Kullback-Leibler divergences, resulting in significant improvements in energy fitting, mode-covering, and stability, with performance gaps increasing with model dimensions.
We propose ratio divergence (RD) learning for discrete energy-based models, a method that utilizes both training data and a tractable target energy function. We apply RD learning to restricted Boltzmann machines (RBMs), which are a minimal model that satisfies the universal approximation theorem for discrete distributions. RD learning combines the strength of both forward and reverse Kullback-Leibler divergence (KLD) learning, effectively addressing the "notorious" issues of underfitting with the forward KLD and mode-collapse with the reverse KLD. Since the summation of forward and reverse KLD seems to be sufficient to combine the strength of both approaches, we include this learning method as a direct baseline in numerical experiments to evaluate its effectiveness. Numerical experiments demonstrate that RD learning significantly outperforms other learning methods in terms of energy function fitting, mode-covering, and learning stability across various discrete energy-based models. Moreover, the performance gaps between RD learning and the other learning methods become more pronounced as the dimensions of target models increase.