N-gram Statistical Stemmer for Bangla Corpus
This work addresses text retrieval enhancement for Bangla language processing, presenting an incremental improvement over existing rule-based methods.
The paper tackled the problem of stemming for Bangla text by proposing an N-gram statistical stemmer that clusters words using dice coefficient, achieving around 87% accuracy in cluster formation.
Stemming is a process that can be utilized to trim inflected words to stem or root form. It is useful for enhancing the retrieval effectiveness, especially for text search in order to solve the mismatch problems. Previous research on Bangla stemming mostly relied on eliminating multiple suffixes from a solitary word through a recursive rule based procedure to recover progressively applicable relative root. Our proposed system has enhanced the aforementioned exploration by actualizing one of the stemming algorithms called N-gram stemming. By utilizing an affiliation measure called dice coefficient, related sets of words are clustered depending on their character structure. The smallest word in one cluster may be considered as the stem. We additionally analyzed Affinity Propagation clustering algorithms with coefficient similarity as well as with median similarity. Our result indicates N-gram stemming techniques to be effective in general which gave us around 87% accurate clusters.