Find the Intention of Instruction: Comprehensive Evaluation of Instruction Understanding for Large Language Models
This addresses the need for better evaluation of LLMs' instruction understanding, which is crucial for their reliable application across various fields, though it is incremental as it builds on existing benchmarks.
The authors tackled the problem of evaluating large language models' instruction-following ability by introducing the Intention of Instruction (IoInst) benchmark, which tests models' focus and comprehension without distraction, finding that even state-of-the-art models still lack this capability.
One of the key strengths of Large Language Models (LLMs) is their ability to interact with humans by generating appropriate responses to given instructions. This ability, known as instruction-following capability, has established a foundation for the use of LLMs across various fields and serves as a crucial metric for evaluating their performance. While numerous evaluation benchmarks have been developed, most focus solely on clear and coherent instructions. However, we have noted that LLMs can become easily distracted by instruction-formatted statements, which may lead to an oversight of their instruction comprehension skills. To address this issue, we introduce the Intention of Instruction (IoInst) benchmark. This benchmark evaluates LLMs' capacity to remain focused and understand instructions without being misled by extraneous instructions. The primary objective of this benchmark is to identify the appropriate instruction that accurately guides the generation of a given context. Our findings suggest that even recently introduced state-of-the-art models still lack instruction understanding capability. Along with the proposition of IoInst in this study, we also present broad analyses of the several strategies potentially applicable to IoInst.