Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction
This work addresses the need for more refined information extraction benchmarks for large language model researchers, though it is incremental as it builds on existing models and datasets.
The paper tackles the problem of adapting information extraction techniques for large language models by creating a fine-grained benchmark dataset with augmented instructions. The results show that encoder-decoder models like T5 generalize better to unseen information types, while ChatGPT adapts better to new task forms, with performance influenced by architecture, data diversity, and learning techniques rather than just model scale.
Information Extraction (IE) is an essential task in Natural Language Processing. Traditional methods have relied on coarse-grained extraction with simple instructions. However, with the emergence of Large Language Models (LLMs), there is a need to adapt IE techniques to leverage the capabilities of these models. This paper introduces a fine-grained IE benchmark dataset tailored for LLMs, employing augmented instructions for each information type, which includes task descriptions, extraction rules, output formats, and examples. Through extensive evaluations, we observe that encoder-decoder models, particularly T5 and FLAN-T5, perform well in generalizing to unseen information types, while ChatGPT exhibits greater adaptability to new task forms. Our results also indicate that performance is not solely dictated by model scale, and highlight the significance of architecture, data diversity, and learning techniques. This work paves the way for a more refined and versatile utilization of LLMs in Information Extraction.