The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models
This work addresses the challenge of assessing sequential instruction following in LLMs, which is crucial for their robustness in real-world applications, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating large language models' ability to follow multiple sequential instructions by introducing the SIFo benchmark, which includes four tasks and shows that recent, larger models significantly outperform older, smaller ones, though all models struggle with instruction sequences.
Following multiple instructions is a crucial ability for large language models (LLMs). Evaluating this ability comes with significant challenges: (i) limited coherence between multiple instructions, (ii) positional bias where the order of instructions affects model performance, and (iii) a lack of objectively verifiable tasks. To address these issues, we introduce a benchmark designed to evaluate models' abilities to follow multiple instructions through sequential instruction following (SIFo) tasks. In SIFo, the successful completion of multiple instructions is verifiable by examining only the final instruction. Our benchmark evaluates instruction following using four tasks (text modification, question answering, mathematics, and security rules), each assessing different aspects of sequential instruction following. Our evaluation of popular LLMs, both closed-source and open-source, shows that more recent and larger models significantly outperform their older and smaller counterparts on the SIFo tasks, validating the benchmark's effectiveness. All models struggle with following sequences of instructions, hinting at an important lack of robustness of today's language models.