ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
This addresses a critical shortcoming in LLMs for users dealing with complex prompts, though it is incremental as it builds on existing instruction-following benchmarks.
The paper tackles the problem of evaluating large language models (LLMs) on detecting and resolving conflicts in user instructions, revealing that while some models like DeepSeek-R1 and Claude-4.5-Sonnet achieve high F1-scores (91.5% and 87.3%), they rarely notify users about conflicts or request clarification.
Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints-a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.