Teaching language models to support answers with verified quotes
This addresses the issue of trustworthiness in AI-generated answers for users by enabling models to cite evidence, though it is incremental as it builds on existing reinforcement learning and open-book QA methods.
The paper tackles the problem of language models hallucinating answers by training a 280 billion parameter model, GopherCite, to generate answers with verified quotes from search engine or user-provided documents, achieving high-quality responses 80% of the time on a Natural Questions subset and 67% on an ELI5 subset, with abstention improving these to 90% and 80% respectively.
Recent large language models often answer factual questions correctly. But users can't trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train "open-book" QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model's response is found to be high-quality 80\% of the time on this Natural Questions subset, and 67\% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90\% and 80\% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows why citation is only one part of an overall strategy for safety and trustworthiness: not all claims supported by evidence are true.