REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting
This addresses fairness in opinion summarization for users of LLMs, offering a practical method without needing hyperparameter tuning or distributional data, though it is incremental as it builds on cognitive science insights.
The study tackled the problem of bias in opinion summarization by large language models (LLMs) by introducing frequency framed prompting (REFER), which improved fairness, especially in larger models and with stronger reasoning instructions.
Individuals express diverse opinions, a fair summary should represent these viewpoints comprehensively. Previous research on fairness in opinion summarisation using large language models (LLMs) relied on hyperparameter tuning or providing ground truth distributional information in prompts. However, these methods face practical limitations: end-users rarely modify default model parameters, and accurate distributional information is often unavailable. Building upon cognitive science research demonstrating that frequency-based representations reduce systematic biases in human statistical reasoning by making reference classes explicit and reducing cognitive load, this study investigates whether frequency framed prompting (REFER) can similarly enhance fairness in LLM opinion summarisation. Through systematic experimentation with different prompting frameworks, we adapted techniques known to improve human reasoning to elicit more effective information processing in language models compared to abstract probabilistic representations.Our results demonstrate that REFER enhances fairness in language models when summarising opinions. This effect is particularly pronounced in larger language models and using stronger reasoning instructions.