Bridging the Data Gap: Creating a Hindi Text Summarization Dataset from the English XSUM
This work addresses the data scarcity problem for Hindi NLP researchers, offering a scalable methodology for low-resource languages, though it is incremental in applying existing translation and adaptation techniques.
This study tackled the lack of high-quality text summarization datasets for Hindi by creating a comprehensive dataset using an automated framework based on the English XSUM corpus, resulting in a diverse, multi-thematic resource that mirrors the original's complexity.
Current advancements in Natural Language Processing (NLP) have largely favored resource-rich languages, leaving a significant gap in high-quality datasets for low-resource languages like Hindi. This scarcity is particularly evident in text summarization, where the development of robust models is hindered by a lack of diverse, specialized corpora. To address this disparity, this study introduces a cost-effective, automated framework for creating a comprehensive Hindi text summarization dataset. By leveraging the English Extreme Summarization (XSUM) dataset as a source, we employ advanced translation and linguistic adaptation techniques. To ensure high fidelity and contextual relevance, we utilize the Crosslingual Optimized Metric for Evaluation of Translation (COMET) for validation, supplemented by the selective use of Large Language Models (LLMs) for curation. The resulting dataset provides a diverse, multi-thematic resource that mirrors the complexity of the original XSUM corpus. This initiative not only provides a direct tool for Hindi NLP research but also offers a scalable methodology for democratizing NLP in other underserved languages. By reducing the costs associated with dataset creation, this work fosters the development of more nuanced, culturally relevant models in computational linguistics.