The Dependence of Frequency Distributions on Multiple Meanings of Words, Codes and Signs
This addresses a problem in linguistics and information theory by showing that coding schemes influence word-frequency distributions, with implications for text analysis and cross-linguistic comparisons.
The study investigated how multiple meanings of words affect frequency distributions by deleting letters from English words, finding that word-frequency distributions are broad and fat-tailed for full words but become exponential when represented by only the first letter, with predictions matching sequences from L=6 to 1 letters. Comparisons with Chinese texts showed similar distributions, indicating that coding differences, not language itself, cause shape variations.
The dependence of the frequency distributions due to multiple meanings of words in a text is investigated by deleting letters. By coding the words with fewer letters the number of meanings per coded word increases. This increase is measured and used as an input in a predictive theory. For a text written in English, the word-frequency distribution is broad and fat-tailed, whereas if the words are only represented by their first letter the distribution becomes exponential. Both distribution are well predicted by the theory, as is the whole sequence obtained by consecutively representing the words by the first L=6,5,4,3,2,1 letters. Comparisons of texts written by Chinese characters and the same texts written by letter-codes are made and the similarity of the corresponding frequency-distributions are interpreted as a consequence of the multiple meanings of Chinese characters. This further implies that the difference of the shape for word-frequencies for an English text written by letters and a Chinese text written by Chinese characters is due to the coding and not to the language per se.