Exploring Language Model Generalization in Low-Resource Extractive QA
This work addresses the problem of domain generalization in extractive QA for researchers and practitioners, highlighting limitations in current LLMs for specialized applications, but it is incremental as it identifies issues without proposing new solutions.
The paper investigated whether large language models can generalize to low-resource domains like medicine and law in extractive question answering without additional training, finding that they struggle with closed-domain demands such as retrieving long answer spans and discriminating word senses, with scaling parameters not always effective.
In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to explain the performance gap empirically. Our findings suggest that: (a) LLMs struggle with dataset demands of closed domains such as retrieving long answer spans; (b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; (c) Scaling model parameters is not always effective for cross domain generalization; and (d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.