The empirical structure of word frequency distributions
This challenges a foundational assumption in linguistics and cognitive science about the role of Zipf's law in communication, potentially shifting focus to geometric distributions for understanding human communicative efficiency.
The study tackled the problem of whether Zipf's law in word frequency distributions optimizes human communication by showing that first names and English nouns/verbs follow geometric distributions when analyzed within communities, with power laws emerging only from aggregation, indicating that efficient communication arises from functional structures meeting information theory constraints, not Zipf's law.
The frequencies at which individual words occur across languages follow power law distributions, a pattern of findings known as Zipf's law. A vast literature argues over whether this serves to optimize the efficiency of human communication, however this claim is necessarily post hoc, and it has been suggested that Zipf's law may in fact describe mixtures of other distributions. From this perspective, recent findings that Sinosphere first (family) names are geometrically distributed are notable, because this is actually consistent with information theoretic predictions regarding optimal coding. First names form natural communicative distributions in most languages, and I show that when analyzed in relation to the communities in which they are used, first name distributions across a diverse set of languages are both geometric and, historically, remarkably similar, with power law distributions only emerging when empirical distributions are aggregated. I then show this pattern of findings replicates in communicative distributions of English nouns and verbs. These results indicate that if lexical distributions support efficient communication, they do so because their functional structures directly satisfy the constraints described by information theory, and not because of Zipf's law. Understanding the function of these information structures is likely to be key to explaining humankind's remarkable communicative capacities.