Native Logical and Hierarchical Representations with Subspace Embeddings
This addresses the limitation of neural embeddings in handling logical and hierarchical reasoning, which is crucial for applications in natural language processing and knowledge representation.
The paper tackles the problem of traditional neural embeddings struggling with higher-level reasoning and asymmetric relationships by introducing a novel paradigm of embedding concepts as linear subspaces, achieving state-of-the-art results in reconstruction and link prediction on WordNet and surpassing bi-encoder baselines on natural language inference benchmarks.
Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel paradigm: embedding concepts as linear subspaces. This framework inherently models generality via subspace dimensionality and hierarchy through subspace inclusion. It naturally supports set-theoretic operations like intersection (conjunction), linear sum (disjunction) and orthogonal complements (negations), aligning with classical formal semantics. To enable differentiable learning, we propose a smooth relaxation of orthogonal projection operators, allowing for the learning of both subspace orientation and dimension. Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet. Furthermore, on natural language inference benchmarks, our subspace embeddings surpass bi-encoder baselines, offering an interpretable formulation of entailment that is both geometrically grounded and amenable to logical operations.