HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs
This addresses the challenge of verifying LLM outputs for users, though it has limitations when LLMs are wrong.
The paper tackles the problem of LLMs hallucinating non-factual statements by proposing Highlighted Chain-of-Thought Prompting (HoT), which prompts LLMs to generate responses with XML tags that ground facts to the input, resulting in outperforming vanilla CoT on 17 tasks and improving human verification accuracy and efficiency when LLMs are correct.
An Achilles heel of Large Language Models (LLMs) is their tendency to hallucinate non-factual statements. A response mixed of factual and non-factual statements poses a challenge for humans to verify and accurately base their decisions on. To combat this problem, we propose Highlighted Chain-of-Thought Prompting (HoT), a technique for prompting LLMs to generate responses with XML tags that ground facts to those provided in the query. That is, given an input question, LLMs would first re-format the question to add XML tags highlighting key facts, and then, generate a response with highlights over the facts referenced from the input. Interestingly, in few-shot settings, HoT outperforms vanilla chain of thought prompting (CoT) on a wide range of 17 tasks from arithmetic, reading comprehension to logical reasoning. When asking humans to verify LLM responses, highlights help time-limited participants to more accurately and efficiently recognize when LLMs are correct. Yet, surprisingly, when LLMs are wrong, HoTs tend to make users believe that an answer is correct.