Estimating the Entropy of Linguistic Distributions
This work addresses a methodological gap for linguists studying language entropy, offering practical guidance to improve accuracy in information-theoretic analyses.
The paper tackled the problem of entropy estimation for linguistic data, finding that poor estimators can overstate effect sizes in linguistic studies, and provided recommendations for better estimation methods.
Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data because researchers do not have access to the underlying probability distribution that gives rise to these data. While entropy estimation is a well-studied problem in other fields, there is not yet a comprehensive exploration of the efficacy of entropy estimators for use with linguistic data. In this work, we fill this void, studying the empirical effectiveness of different entropy estimators for linguistic distributions. In a replication of two recent information-theoretic linguistic studies, we find evidence that the reported effect size is over-estimated due to over-reliance on poor entropy estimators. Finally, we end our paper with concrete recommendations for entropy estimation depending on distribution type and data availability.