Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation
This addresses the need for reliable and accurate LLM evaluations in clinical settings, though it is incremental as it builds on existing methods with a tailored approach.
The paper tackles the problem of misalignment in LLM-based medical evaluation by introducing HDCEval, a hierarchical framework that decomposes tasks into subtasks evaluated by expert models, resulting in significant improvement in alignment with human evaluators.
In the rapidly evolving landscape of large language models (LLMs) for medical applications, ensuring the reliability and accuracy of these models in clinical settings is paramount. Existing benchmarks often focus on fixed-format tasks like multiple-choice QA, which fail to capture the complexity of real-world clinical diagnostics. Moreover, traditional evaluation metrics and LLM-based evaluators struggle with misalignment, often providing oversimplified assessments that do not adequately reflect human judgment. To address these challenges, we introduce HDCEval, a Hierarchical Divide-and-Conquer Evaluation framework tailored for fine-grained alignment in medical evaluation. HDCEval is built on a set of fine-grained medical evaluation guidelines developed in collaboration with professional doctors, encompassing Patient Question Relevance, Medical Knowledge Correctness, and Expression. The framework decomposes complex evaluation tasks into specialized subtasks, each evaluated by expert models trained through Attribute-Driven Token Optimization (ADTO) on a meticulously curated preference dataset. This hierarchical approach ensures that each aspect of the evaluation is handled with expert precision, leading to a significant improvement in alignment with human evaluators.