LLM Benchmark-User Need Misalignment for Climate Change
This addresses the problem of benchmark misalignment for climate change applications, providing actionable guidance for developers and researchers, though it is incremental in refining evaluation methods.
The study identified a significant mismatch between existing LLM benchmarks and real-world user needs for climate change information, while finding that human-AI knowledge interactions closely resemble human-human patterns.
Climate change is a major socio-scientific issue shapes public decision-making and policy discussions. As large language models (LLMs) increasingly serve as an interface for accessing climate knowledge, whether existing benchmarks reflect user needs is critical for evaluating LLM in real-world settings. We propose a Proactive Knowledge Behaviors Framework that captures the different human-human and human-AI knowledge seeking and provision behaviors. We further develop a Topic-Intent-Form taxonomy and apply it to analyze climate-related data representing different knowledge behaviors. Our results reveal a substantial mismatch between current benchmarks and real-world user needs, while knowledge interaction patterns between humans and LLMs closely resemble those in human-human interactions. These findings provide actionable guidance for benchmark design, RAG system development, and LLM training. Code is available at https://github.com/OuchengLiu/LLM-Misalign-Climate-Change.