Switch Spaces: Learning Product Spaces with Sparse Gating
This work addresses the need for efficient and effective representation learning in domains like knowledge graphs and recommendations, though it appears incremental as it builds on existing product space methods.
The paper tackled the problem of learning embedding spaces with appropriate geometric inductive biases for representation learning by proposing Switch Spaces, a data-driven approach that uses sparse gating to dynamically combine Euclidean and non-Euclidean manifolds, achieving new state-of-the-art performances in knowledge graph completion and item recommendations.
Learning embedding spaces of suitable geometry is critical for representation learning. In order for learned representations to be effective and efficient, it is ideal that the geometric inductive bias aligns well with the underlying structure of the data. In this paper, we propose Switch Spaces, a data-driven approach for learning representations in product space. Specifically, product spaces (or manifolds) are spaces of mixed curvature, i.e., a combination of multiple euclidean and non-euclidean (hyperbolic, spherical) manifolds. To this end, we introduce sparse gating mechanisms that learn to choose, combine and switch spaces, allowing them to be switchable depending on the input data with specialization. Additionally, the proposed method is also efficient and has a constant computational complexity regardless of the model size. Experiments on knowledge graph completion and item recommendations show that the proposed switch space achieves new state-of-the-art performances, outperforming pure product spaces and recently proposed task-specific models.