Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models
This addresses data quality issues in multilingual NLP applications, though it is incremental as it builds on existing embedding and translation techniques.
This paper tackled multilingual text deduplication by comparing a two-step translation-embedding method with a multilingual embedding model, finding the two-step approach achieved higher F1 scores (82% vs. 60%), which could be increased to 89% with domain-specific rules.
This paper addresses the deduplication of multilingual textual data using advanced NLP tools. We compare a two-step method involving translation to English followed by embedding with mpnet, and a multilingual embedding model (distiluse). The two-step approach achieved a higher F1 score (82% vs. 60%), particularly with less widely used languages, which can be increased up to 89% by leveraging expert rules based on domain knowledge. We also highlight limitations related to token length constraints and computational efficiency. Our methodology suggests improvements for future multilingual deduplication tasks.