Representation Learning via Variational Bayesian Networks
This work addresses data scarcity for entities in the long-tail, which is a common issue in domains like linguistics, recommendations, and medical inference, but it is incremental as it builds on Bayesian and variational methods.
The paper tackles the problem of learning representations for entities with scarce data, particularly in the long-tail, by introducing Variational Bayesian Networks (VBN) that use hierarchical priors and relational information to propagate knowledge and model uncertainty as densities. The results show that VBN outperforms existing methods across multiple datasets, especially in the long-tail.
We present Variational Bayesian Network (VBN) - a novel Bayesian entity representation learning model that utilizes hierarchical and relational side information and is particularly useful for modeling entities in the ``long-tail'', where the data is scarce. VBN provides better modeling for long-tail entities via two complementary mechanisms: First, VBN employs informative hierarchical priors that enable information propagation between entities sharing common ancestors. Additionally, VBN models explicit relations between entities that enforce complementary structure and consistency, guiding the learned representations towards a more meaningful arrangement in space. Second, VBN represents entities by densities (rather than vectors), hence modeling uncertainty that plays a complementary role in coping with data scarcity. Finally, we propose a scalable Variational Bayes optimization algorithm that enables fast approximate Bayesian inference. We evaluate the effectiveness of VBN on linguistic, recommendations, and medical inference tasks. Our findings show that VBN outperforms other existing methods across multiple datasets, and especially in the long-tail.