Compositional Structures in Neural Embedding and Interaction Decompositions
This work addresses a foundational gap in understanding representation learning for the machine learning community, though it appears incremental as it builds on existing formalizations.
The paper tackles the problem of formally explaining the emergence of structural patterns in neural network embeddings by establishing a correspondence between linear algebraic structures in vector embeddings and conditional independence constraints in probability distributions, introducing a characterization of compositional structures through interaction decompositions and providing necessary and sufficient conditions for their presence.
We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks and conditional independence constraints on the probability distributions modeled by these networks. Our framework aims to shed light on the emergence of structural patterns in data representations, a phenomenon widely acknowledged but arguably still lacking a solid formal grounding. Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.