Clustering of Spell Variations for Proper Nouns Transliterated from the other languages
This work addresses a domain-specific issue in NLP for text data processing, particularly for applications involving Indian language proper nouns, and is incremental as it applies existing ML techniques to a known bottleneck.
The paper tackled the problem of spell variations in proper nouns transliterated from Indian languages, which complicates NLP tasks like address and name processing, by proposing a clustering method using Affinity Propagation and similarity thresholds to reduce variations, achieving a considerable reduction in spell variations.
One of the prominent problems with processing and operating on text data is the non uniformity of it. Due to the change in the dialects and languages, the caliber of translation is low. This creates a unique problem while using NLP in text data; which is the spell variation arising from the inconsistent translations and transliterations. This problem can also be further aggravated by the human error arising from the various ways to write a Proper Noun from an Indian language into its English equivalent. Translating proper nouns originating from Indian languages can be complicated as some proper nouns are also used as common nouns which might be taken literally. Applications of NLP that require addresses, names and other proper nouns face this problem frequently. We propose a method to cluster these spell variations for proper nouns using ML techniques and mathematical similarity equations. We aimed to use Affinity Propagation to determine relative similarity between the tokens. The results are augmented by filtering the token-variation pair by a similarity threshold. We were able to reduce the spell variations by a considerable amount. This application can significantly reduce the amount of human annotation efforts needed for data cleansing and formatting.