Scaling Laws in Human Language
This refines understanding of Zipf's and Heaps' laws in linguistics, addressing a specific discrepancy in language systems.
The paper tackled the problem of why Zipf's law on word frequency holds for some languages like English but not for others like Chinese, by proposing a model that accounts for finite vocabulary size, and found that the frequency distribution follows a power law with exponent 1, leading to divergence in Zipf's exponent, and the number of distinct words grows in three stages: linearly, logarithmically, and then saturates.
Zipf's law on word frequency is observed in English, French, Spanish, Italian, and so on, yet it does not hold for Chinese, Japanese or Korean characters. A model for writing process is proposed to explain the above difference, which takes into account the effects of finite vocabulary size. Experiments, simulations and analytical solution agree well with each other. The results show that the frequency distribution follows a power law with exponent being equal to 1, at which the corresponding Zipf's exponent diverges. Actually, the distribution obeys exponential form in the Zipf's plot. Deviating from the Heaps' law, the number of distinct words grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates. This work refines previous understanding about Zipf's law and Heaps' law in language systems.