A Crucial Parameter for Rank-Frequency Relation in Natural Languages
This work provides an incremental improvement in linguistic modeling for researchers in computational linguistics and natural language processing.
The authors tackled the problem of modeling rank-frequency relations in natural languages, showing that the parameter γ is crucial in a refined power law formulation, with empirical evidence indicating it captures resistance to vocabulary growth.
$f \propto r^{-α} \cdot (r+γ)^{-β}$ has been empirically shown more precise than a naïve power law $f\propto r^{-α}$ to model the rank-frequency ($r$-$f$) relation of words in natural languages. This work shows that the only crucial parameter in the formulation is $γ$, which depicts the resistance to vocabulary growth on a corpus. A method of parameter estimation by searching an optimal $γ$ is proposed, where a ``zeroth word'' is introduced technically for the calculation. The formulation and parameters are further discussed with several case studies.