Decision-Oriented Text Evaluation
This addresses the need for better evaluation methods in natural language generation for decision-making contexts, though it is incremental as it builds on existing frameworks by focusing on synergy.
The paper tackled the problem of evaluating generated text in high-stakes domains by proposing a decision-oriented framework that measures its influence on human and LLM decision outcomes, finding that collaborative human-LLM teams significantly outperform individual baselines when using richer analytical commentaries.
Natural language generation (NLG) is increasingly deployed in high-stakes domains, yet common intrinsic evaluation methods, such as n-gram overlap or sentence plausibility, weakly correlate with actual decision-making efficacy. We propose a decision-oriented framework for evaluating generated text by directly measuring its influence on human and large language model (LLM) decision outcomes. Using market digest texts--including objective morning summaries and subjective closing-bell analyses--as test cases, we assess decision quality based on the financial performance of trades executed by human investors and autonomous LLM agents informed exclusively by these texts. Our findings reveal that neither humans nor LLM agents consistently surpass random performance when relying solely on summaries. However, richer analytical commentaries enable collaborative human-LLM teams to outperform individual human or agent baselines significantly. Our approach underscores the importance of evaluating generated text by its ability to facilitate synergistic decision-making between humans and LLMs, highlighting critical limitations of traditional intrinsic metrics.