IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations
This work addresses the challenge of scalable evaluation for LLMs in expert-driven domains like education, offering a practical tool for responsible deployment, though it is incremental in improving existing benchmarking approaches.
The paper tackled the problem of evaluating large language models (LLMs) on open-ended interpretive annotation tasks, such as thematic analysis and feedback generation, by introducing IDEAlign, a benchmarking paradigm that uses a 'pick-the-odd-one-out' triplet judgment task to measure similarity to expert human annotations. The results showed that IDEAlign improved alignment with expert judgments by 9-30% compared to traditional metrics, establishing it as a scalable evaluation method.
Large language models (LLMs) are increasingly applied to open-ended, interpretive annotation tasks, such as thematic analysis by researchers or generating feedback on student work by teachers. These tasks involve free-text annotations requiring expert-level judgments grounded in specific objectives (e.g., research questions or instructional goals). Evaluating whether LLM-generated annotations align with those generated by expert humans is challenging to do at scale, and currently, no validated, scalable measure of similarity in ideas exists. In this paper, we (i) introduce the scalable evaluation of interpretive annotation by LLMs as a critical and understudied task, (ii) propose IDEAlgin, an intuitive benchmarking paradigm for capturing expert similarity ratings via a "pick-the-odd-one-out" triplet judgment task, and (iii) evaluate various similarity metrics, including vector-based ones (topic models, embeddings) and LLM-as-a-judge via IDEAlgin, against these human benchmarks. Applying this approach to two real-world educational datasets (interpretive analysis and feedback generation), we find that vector-based metrics largely fail to capture the nuanced dimensions of similarity meaningful to experts. Prompting LLMs via IDEAlgin significantly improves alignment with expert judgments (9-30% increase) compared to traditional lexical and vector-based metrics. These results establish IDEAlgin as a promising paradigm for evaluating LLMs against open-ended expert annotations at scale, informing responsible deployment of LLMs in education and beyond.