InFoBench: Evaluating Instruction Following Ability in Large Language Models
This addresses the need for better evaluation methods in AI research, though it is incremental as it builds on existing instruction-following assessment.
The paper tackled the problem of evaluating large language models' ability to follow instructions by introducing the Decomposed Requirements Following Ratio (DRFR) metric and InFoBench benchmark, finding that DRFR is more reliable and GPT-4 serves as a cost-efficient annotator.
This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions. Addressing a gap in current methodologies, DRFR breaks down complex instructions into simpler criteria, facilitating a detailed analysis of LLMs' compliance with various aspects of tasks. Alongside this metric, we present InFoBench, a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. Our experiments compare DRFR with traditional scoring methods and explore annotation sources, including human experts, crowd-sourced workers, and GPT-4. The findings demonstrate DRFR's higher reliability and the effectiveness of using GPT-4 as a cost-efficient annotator. The evaluation of several advanced LLMs using this framework reveals their strengths and areas needing improvement, particularly in complex instruction-following. This study contributes a novel metric and benchmark, offering insights for future LLM development and evaluation.