Semantic Holism and Word Representations in Artificial Neural Networks
This work addresses a theoretical problem for researchers in natural language processing and philosophy of language, but it is incremental as it builds on existing models without introducing new methods or data.
The paper tackles the problem of explaining semantic properties in word representations from neural networks like Skip-ram, proposing a Frege-inspired holistic approach as an alternative to the distributional hypothesis, with no concrete numerical results reported.
Artificial neural networks are a state-of-the-art solution for many problems in natural language processing. What can we learn about language and meaning from the way artificial neural networks represent it? Word representations obtained from the Skip-gram variant of the word2vec model exhibit interesting semantic properties. This is usually explained by referring to the general distributional hypothesis, which states that the meaning of the word is given by the contexts where it occurs. We propose a more specific approach based on Frege's holistic and functional approach to meaning. Taking Tugendhat's formal reinterpretation of Frege's work as a starting point, we demonstrate that it is analogical to the process of training the Skip-gram model and offers a possible explanation of its semantic properties.