Zipf's law holds for phrases, not words
This work addresses a fundamental problem in linguistics and text analysis by revealing that phrases, not words, better follow Zipf's law, which could impact fields like natural language processing and information theory.
The authors tackled the limited applicability of Zipf's law to words in human language by showing that it holds for phrases over nine orders of magnitude, compared to only three to four for words, using a statistical mechanical method for text partitioning.
With Zipf's law being originally and most famously observed for word frequency, it is surprisingly limited in its applicability to human language, holding over no more than three to four orders of magnitude before hitting a clear break in scaling. Here, building on the simple observation that phrases of one or more words comprise the most coherent units of meaning in language, we show empirically that Zipf's law for phrases extends over as many as nine orders of rank magnitude. In doing so, we develop a principled and scalable statistical mechanical method of random text partitioning, which opens up a rich frontier of rigorous text analysis via a rank ordering of mixed length phrases.