Generalists vs. Specialists: Evaluating Large Language Models for Urdu
This addresses the underrepresentation of Urdu, a low-resource language with 70 million speakers, in NLP by evaluating model performance, though it is incremental as it applies existing methods to a new context.
The paper compared general-purpose and fine-tuned special-purpose large language models on Urdu tasks, finding that special-purpose models consistently outperformed general-purpose ones, and GPT-4-Turbo's evaluations aligned more closely with human judgments than Llama-3-8b's.
In this paper, we compare general-purpose models, GPT-4-Turbo and Llama-3-8b, with special-purpose models--XLM-Roberta-large, mT5-large, and Llama-3-8b--that have been fine-tuned on specific tasks. We focus on seven classification and seven generation tasks to evaluate the performance of these models on Urdu language. Urdu has 70 million native speakers, yet it remains underrepresented in Natural Language Processing (NLP). Despite the frequent advancements in Large Language Models (LLMs), their performance in low-resource languages, including Urdu, still needs to be explored. We also conduct a human evaluation for the generation tasks and compare the results with the evaluations performed by GPT-4-Turbo, Llama-3-8b and Claude 3.5 Sonnet. We find that special-purpose models consistently outperform general-purpose models across various tasks. We also find that the evaluation done by GPT-4-Turbo for generation tasks aligns more closely with human evaluation compared to the evaluation the evaluation done by Llama-3-8b. This paper contributes to the NLP community by providing insights into the effectiveness of general and specific-purpose LLMs for low-resource languages.