Exploiting Primacy Effect To Improve Large Language Models
This addresses bias issues in LLMs for NLP applications, offering a method to turn biases into opportunities for improvement, though it appears incremental as it builds on known biases.
The study tackled the problem of positional biases in fine-tuned Large Language Models (LLMs), specifically the primacy effect in Multiple Choice Question Answering (MCQA), by strategically reordering answer options based on semantic similarity to the query. This approach significantly improved performance in MCQA without requiring knowledge of the correct answer.
Large Language Models (LLMs) have become essential in many Natural Language Processing (NLP) tasks, leveraging extensive pre-training and fine-tuning to achieve high accuracy. However, like humans, LLMs exhibit biases, particularly positional biases such as primacy and recency effects, which can influence the accuracy of the answers. The primacy effect-where items presented first are more likely to be remembered or selected-plays a key role in Multiple Choice Question Answering (MCQA), where the order of answer options can affect prediction outcomes. This study focuses on primacy bias in fine-tuned LLMs: We first show that fine-tuning amplifies this bias, probably due to exposure to human-like patterns. Hence, we strategically leverage this effect by reordering response options based on semantic similarity to the query, without requiring knowledge of the correct answer. Our experimental results show that this approach significantly improves performance in MCQA. More generally, our findings underscore the dual nature of biases as both challenges and opportunities, offering insights for bias-aware model design and NLP applications.