Compositional Fusion of Signals in Data Embedding
This work provides insights into the compositional nature of embeddings, which is incremental for researchers and practitioners in AI and machine learning.
The paper tackled the problem of understanding how embeddings fuse multiple signals in real-world data, finding that word embeddings combine semantic and morphological signals, BERT sentence embeddings decompose into word vectors, and user embeddings in recommender systems exhibit demographic signals without explicit training.
Embeddings in AI convert symbolic structures into fixed-dimensional vectors, effectively fusing multiple signals. However, the nature of this fusion in real-world data is often unclear. To address this, we introduce two methods: (1) Correlation-based Fusion Detection, measuring correlation between known attributes and embeddings, and (2) Additive Fusion Detection, viewing embeddings as sums of individual vectors representing attributes. Applying these methods, word embeddings were found to combine semantic and morphological signals. BERT sentence embeddings were decomposed into individual word vectors of subject, verb and object. In the knowledge graph-based recommender system, user embeddings, even without training on demographic data, exhibited signals of demographics like age and gender. This study highlights that embeddings are fusions of multiple signals, from Word2Vec components to demographic hints in graph embeddings.