Transparency Helps Reveal When Language Models Learn Meaning
This work addresses the problem of understanding when language models acquire meaning for NLP researchers, providing evidence that current models struggle with context-dependent semantics, which is incremental in building on prior critiques.
The paper investigates conditions under which language models learn meaning, finding that models successfully emulate semantic relations with context-independent languages but degrade with context-dependent ones, and experiments on natural language show current models fail to represent semantics well due to context-dependence.
Many current NLP systems are built from language models trained to optimize unsupervised objectives on large amounts of raw text. Under what conditions might such a procedure acquire meaning? Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations (i.e., languages with strong transparency), both autoregressive and masked language models successfully learn to emulate semantic relations between expressions. However, when denotations are changed to be context-dependent with the language otherwise unmodified, this ability degrades. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not represent natural language semantics well. We show this failure relates to the context-dependent nature of natural language form-meaning mappings.