Enhancing Annotated Bibliography Generation with LLM Ensembles
This work addresses the challenge of automating complex scholarly tasks like annotated bibliography generation for researchers and academics, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of generating annotated bibliographies by using an ensemble of Large Language Models (LLMs) in different roles, such as text generation, evaluation, and summarization. The result is a 38% improvement in annotation quality and a 51% reduction in content redundancy compared to individual LLM responses.
This work proposes a novel approach to enhancing annotated bibliography generation through Large Language Model (LLM) ensembles. In particular, multiple LLMs in different roles -- controllable text generation, evaluation, and summarization -- are introduced and validated using a systematic methodology to enhance model performance in scholarly tasks. Output diversity among the ensemble that generates text is obtained using different LLM parameters, followed by an LLM acting as a judge to assess relevance, accuracy, and coherence. Responses selected by several combining strategies are then merged and refined through summarization and redundancy removal techniques. The preliminary experimental validation demonstrates that the combined outputs from the LLM ensemble improve coherence and relevance compared to individual responses, leading to a 38% improvement in annotation quality and a 51% reduction in content redundancy, thus highlighting the potential for automating complex scholarly tasks while maintaining high-quality standards.