MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks
This work addresses the inefficiency of LLMs for IE tasks by providing a more effective and efficient meta-model for few-shot adaptation, though it is incremental as it builds on existing distillation and meta-learning techniques.
The paper tackles the problem of information extraction (IE) tasks where prompting large language models (LLMs) underperforms small models tuned on small datasets, by proposing MetaIE, a framework that distills a small meta-model from an LLM to learn 'important information' as label-to-span matching, enabling effective few-shot adaptation across 6 IE tasks and outperforming other meta-models on 13 datasets.
Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.