Information Theory of Meaningful Communication
This work addresses a foundational issue in information theory for linguistics and AI, offering a novel perspective on how to measure meaningful information in communication.
The authors tackled the problem of quantifying meaningful communication by proposing that information in language should be measured in bits per clause rather than per character, leveraging large language models to estimate this metric.
In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause.