Non-Zipfian Distribution of Stopwords and Subset Selection Models
This work provides a new statistical model for understanding the distribution and selection of stopwords, which could be useful for researchers and practitioners in natural language processing (NLP) for tasks like text preprocessing and information retrieval.
This paper investigates the rank-frequency distribution of stopwords, finding that they follow a Beta Rank Function (BRF) rather than Zipf's law, while non-stopwords are better fitted by a quadratic function. Based on these observations, the authors propose a stopword selection model using a decreasing Hill's function for selection probability, which is validated by direct estimation and analytically shown to produce the BRF for stopwords and explain the quadratic fit for non-stopwords.
Stopwords are words that are not very informative to the content or the meaning of a language text. Most stopwords are function words but can also be common verbs, adjectives and adverbs. In contrast to the well known Zipf's law for rank-frequency plot for all words, the rank-frequency plot for stopwords are best fitted by the Beta Rank Function (BRF). On the other hand, the rank-frequency plots of non-stopwords also deviate from the Zipf's law, but are fitted better by a quadratic function of log-token-count over log-rank than by BRF. Based on the observed rank of stopwords in the full word list, we propose a stopword (subset) selection model that the probability for being selected as a function of the word's rank $r$ is a decreasing Hill's function ($1/(1+(r/r_{mid})^γ)$); whereas the probability for not being selected is the standard Hill's function ( $1/(1+(r_{mid}/r)^γ)$). We validate this selection probability model by a direct estimation from an independent collection of texts. We also show analytically that this model leads to a BRF rank-frequency distribution for stopwords when the original full word list follows the Zipf's law, as well as explaining the quadratic fitting function for the non-stopwords.