ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding
This addresses the problem of evaluating LLMs on complex, multi-source intent understanding for researchers and developers, though it is incremental as it focuses on creating a new benchmark rather than advancing model capabilities.
The authors tackled the lack of a large-scale benchmark for evaluating language models on real-world human intent understanding by introducing ConsintBench, the first dynamic, live evaluation benchmark for consumer intent, which is the largest and most diverse of its kind with real-time updates and automated curation.
Understanding human intent is a complex, high-level task for large language models (LLMs), requiring analytical reasoning, contextual interpretation, dynamic information aggregation, and decision-making under uncertainty. Real-world public discussions, such as consumer product discussions, are rarely linear or involve a single user. Instead, they are characterized by interwoven and often conflicting perspectives, divergent concerns, goals, emotional tendencies, as well as implicit assumptions and background knowledge about usage scenarios. To accurately understand such explicit public intent, an LLM must go beyond parsing individual sentences; it must integrate multi-source signals, reason over inconsistencies, and adapt to evolving discourse, similar to how experts in fields like politics, economics, or finance approach complex, uncertain environments. Despite the importance of this capability, no large-scale benchmark currently exists for evaluating LLMs on real-world human intent understanding, primarily due to the challenges of collecting real-world public discussion data and constructing a robust evaluation pipeline. To bridge this gap, we introduce \bench, the first dynamic, live evaluation benchmark specifically designed for intent understanding, particularly in the consumer domain. \bench is the largest and most diverse benchmark of its kind, supporting real-time updates while preventing data contamination through an automated curation pipeline.