Native Language Identification with Large Language Models
This work addresses the task of predicting a writer's first language from second-language writings, with applications in second language acquisition and forensic linguistics, and is incremental as it applies existing LLMs to a new domain.
The paper tackled the problem of Native Language Identification (NLI) by using large language models like GPT-4, achieving a new performance record of 91.7% on the TOEFL11 test set in a zero-shot setting.
We present the first experiments on Native Language Identification (NLI) using LLMs such as GPT-4. NLI is the task of predicting a writer's first language by analyzing their writings in a second language, and is used in second language acquisition and forensic linguistics. Our results show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes, which has practical implications for real-world applications. Finally, we also show that LLMs can provide justification for their choices, providing reasoning based on spelling errors, syntactic patterns, and usage of directly translated linguistic patterns.