When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following
This addresses the need for efficient evaluation of LLMs in real-world scenarios where multiple instructions are common, though it is incremental in benchmarking and estimation methods.
The paper tackled the problem of evaluating large language models' ability to follow multiple instructions simultaneously by introducing two benchmarks, ManyIFEval and StyleMBPP, and found that performance degrades as instruction count increases, with a logistic regression model predicting performance with about 10% error using modest sample sizes.
As large language models (LLMs) are increasingly applied to real-world scenarios, it becomes crucial to understand their ability to follow multiple instructions simultaneously. To systematically evaluate these capabilities, we introduce two specialized benchmarks for fundamental domains where multiple instructions following is important: Many Instruction-Following Eval (ManyIFEval) for text generation with up to ten instructions, and Style-aware Mostly Basic Programming Problems (StyleMBPP) for code generation with up to six instructions. Our experiments with the created benchmarks across ten LLMs reveal that performance consistently degrades as the number of instructions increases. Furthermore, given the fact that evaluating all the possible combinations of multiple instructions is computationally impractical in actual use cases, we developed three types of regression models that can estimate performance on both unseen instruction combinations and different numbers of instructions which are not used during training. We demonstrate that a logistic regression model using instruction count as an explanatory variable can predict performance of following multiple instructions with approximately 10% error, even for unseen instruction combinations. We show that relatively modest sample sizes (500 for ManyIFEval and 300 for StyleMBPP) are sufficient for performance estimation, enabling efficient evaluation of LLMs under various instruction combinations.