LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Models
This is an incremental improvement for researchers using writing assistance tools, enhancing accuracy and contextual relevance in reference synthesis.
The paper tackles the problem of inaccurate reference handling in technical writing with LLMs by introducing LLM-Ref, a tool that retrieves and generates content directly from text paragraphs, achieving a 3.25x to 6.26x increase in Ragas score compared to baseline RAG systems.
Large Language Models (LLMs) excel in data synthesis but can be inaccurate in domain-specific tasks, which retrieval-augmented generation (RAG) systems address by leveraging user-provided data. However, RAGs require optimization in both retrieval and generation stages, which can affect output quality. In this paper, we present LLM-Ref, a writing assistant tool that aids researchers in writing articles from multiple source documents with enhanced reference synthesis and handling capabilities. Unlike traditional RAG systems that use chunking and indexing, our tool retrieves and generates content directly from text paragraphs. This method facilitates direct reference extraction from the generated outputs, a feature unique to our tool. Additionally, our tool employs iterative response generation, effectively managing lengthy contexts within the language model's constraints. Compared to baseline RAG-based systems, our approach achieves a $3.25\times$ to $6.26\times$ increase in Ragas score, a comprehensive metric that provides a holistic view of a RAG system's ability to produce accurate, relevant, and contextually appropriate responses. This improvement shows our method enhances the accuracy and contextual relevance of writing assistance tools.