A Measure-Theoretic Characterization of Tight Language Models
This addresses a theoretical issue in natural language processing for researchers and practitioners, but it is incremental as it builds on previous characterizations.
The paper tackles the problem of probability mass leakage onto infinite sequences in language models by providing a measure-theoretic characterization, proving that many popular language model families are tight and thus do not leak.
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 over all finite strings. However, in some pathological cases, probability mass can ``leak'' onto the set of infinite sequences. In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense. We also generalize characterizations of tightness proposed in previous works.