Citekit: A Modular Toolkit for Large Language Model Citation Generation
This work addresses a reproducibility and assessment problem for researchers and developers working on verifiable LLM outputs, though it is incremental as it builds on existing methods.
The paper tackles the lack of a unified framework for comparing citation generation methods in LLM-based QA tasks by introducing Citekit, an open-source modular toolkit that facilitates implementation and evaluation, and proposes a new method, self-RAG, which achieves balanced answer accuracy and citation quality.
Enabling Large Language Models (LLMs) to generate citations in Question-Answering (QA) tasks is an emerging paradigm aimed at enhancing the verifiability of their responses when LLMs are utilizing external references to generate an answer. However, there is currently no unified framework to standardize and fairly compare different citation generation methods, leading to difficulties in reproducing different methods and a comprehensive assessment. To cope with the problems above, we introduce \name, an open-source and modular toolkit designed to facilitate the implementation and evaluation of existing citation generation methods, while also fostering the development of new approaches to improve citation quality in LLM outputs. This tool is highly extensible, allowing users to utilize 4 main modules and 14 components to construct a pipeline, evaluating an existing method or innovative designs. Our experiments with two state-of-the-art LLMs and 11 citation generation baselines demonstrate varying strengths of different modules in answer accuracy and citation quality improvement, as well as the challenge of enhancing granularity. Based on our analysis of the effectiveness of components, we propose a new method, self-RAG \snippet, obtaining a balanced answer accuracy and citation quality. Citekit is released at https://github.com/SjJ1017/Citekit.