Everything is Plausible: Investigating the Impact of LLM Rationales on Human Notions of Plausibility
This research highlights a novel application of LLMs for studying human cognition, but raises concerns about their potential to sway beliefs even in expert domains like common sense.
The study investigated how LLM-generated rationales influence human plausibility judgments on commonsense benchmarks, finding that PRO rationales increased and CON rationales decreased mean human ratings, indicating humans find these rationales convincing.
We investigate the degree to which human plausibility judgments of multiple-choice commonsense benchmark answers are subject to influence by (im)plausibility arguments for or against an answer, in particular, using rationales generated by LLMs. We collect 3,000 plausibility judgments from humans and another 13,600 judgments from LLMs. Overall, we observe increases and decreases in mean human plausibility ratings in the presence of LLM-generated PRO and CON rationales, respectively, suggesting that, on the whole, human judges find these rationales convincing. Experiments with LLMs reveal similar patterns of influence. Our findings demonstrate a novel use of LLMs for studying aspects of human cognition, while also raising practical concerns that, even in domains where humans are ``experts'' (i.e., common sense), LLMs have the potential to exert considerable influence on people's beliefs.