On Psychology of AI -- Does Primacy Effect Affect ChatGPT and Other LLMs?
This research addresses the problem of cognitive biases in AI systems for users and developers, though it is incremental as it applies a known psychological effect to LLMs without proposing new methods.
The study investigated whether the primacy effect, a psychological bias where earlier information influences decisions, affects large language models (LLMs) like ChatGPT, Gemini, and Claude by adapting Asch's experiment with 200 candidate pairs. Results showed that in simultaneous prompts, ChatGPT preferred candidates with positive adjectives first, while Gemini showed no preference and Claude refused to choose; in separate prompts, ChatGPT and Claude often gave equal ratings but otherwise preferred negative-first candidates, and Gemini consistently preferred negative-first candidates.
We study the primacy effect in three commercial LLMs: ChatGPT, Gemini and Claude. We do this by repurposing the famous experiment Asch (1946) conducted using human subjects. The experiment is simple, given two candidates with equal descriptions which one is preferred if one description has positive adjectives first before negative ones and another description has negative adjectives followed by positive ones. We test this in two experiments. In one experiment, LLMs are given both candidates simultaneously in the same prompt, and in another experiment, LLMs are given both candidates separately. We test all the models with 200 candidate pairs. We found that, in the first experiment, ChatGPT preferred the candidate with positive adjectives listed first, while Gemini preferred both equally often. Claude refused to make a choice. In the second experiment, ChatGPT and Claude were most likely to rank both candidates equally. In the case where they did not give an equal rating, both showed a clear preference to a candidate that had negative adjectives listed first. Gemini was most likely to prefer a candidate with negative adjectives listed first.