Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
This study addresses the problem of overreliance on large language models for users who rely on these models for information, particularly in situations where accuracy is crucial.
The study found that explanations increase reliance on large language model responses, but providing sources or highlighting inconsistencies can mitigate overreliance on incorrect responses, with a sample size of 308 participants. The results show that the presence of explanations, sources, and inconsistencies shape users' reliance on LLM responses.
Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.