Leveraging Open-Source Large Language Models for Native Language Identification
This addresses the need for cost-effective and transparent NLI tools for applications in forensics, marketing, and language acquisition, though it is incremental as it adapts existing methods.
The study tackled the problem of Native Language Identification (NLI) by evaluating open-source large language models (LLMs) and found that while they underperform closed-source LLMs out-of-the-box, fine-tuning them achieves comparable performance.
Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.