Stolen Probability: A Structural Weakness of Neural Language Models
This addresses a fundamental limitation in neural language models that could affect their fairness and performance across various NLP applications, though it is incremental in highlighting a specific structural issue.
The paper identifies a structural weakness in neural language models where the dot-product distance metric leads to a suboptimal ordering of the embedding space, causing some words to be impoverished in probability assignment at the expense of others, with theoretical and empirical analyses showing bounded probabilities for interior words.
Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.