Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ
This work addresses the problem of evaluating LLMs in multi-turn conversations for developers and researchers, showing that static benchmarks are insufficient for identifying deployment-relevant issues, though it is incremental in extending existing evaluation methods.
The study investigated whether conversation length affects the truthfulness of LLM responses by evaluating three LLMs on the BoolQ dataset under different length and scaffolding conditions, revealing model-specific vulnerabilities that were not detectable in single-turn tests.
Single-prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting.