Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation
This addresses the challenge of costly expert annotations for LLM fine-tuning, offering a more robust approach for tasks requiring high-quality response selection.
The paper tackles the problem of customizing LLMs for specific tasks by developing a method to rank candidate responses using partial ordering instead of full ordering, which improves response generation and task accuracy on benchmark datasets like textual entailment and multi-document question answering.
Customizing LLMs for a specific task involves separating high-quality responses from lower-quality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system's improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named Rescue, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.