Uncovering Biases with Reflective Large Language Models
This addresses the issue of propagating biases in supervised learning for AI developers and researchers, though it appears incremental as it builds on existing LLM methods for bias detection.
The paper tackles the problem of biases and errors in human-labeled data for machine learning by introducing the Reflective LLM Dialogue Framework (RLDF), which uses adversarial dialogues between LLMs to detect and correct inconsistencies, resulting in successful identification of biases in public content and exposure of data limitations.
Biases and errors in human-labeled data present significant challenges for machine learning, especially in supervised learning reliant on potentially flawed ground truth data. These flaws, including diagnostic errors and societal biases, risk being propagated and amplified through models trained using maximum likelihood estimation. We present the Reflective LLM Dialogue Framework RLDF, which leverages structured adversarial dialogues between multiple instances of a single LLM or different LLMs to uncover diverse perspectives and correct inconsistencies. By conditioning LLMs to adopt opposing stances, RLDF enables systematic bias detection through conditional statistics, information theory, and divergence metrics. Experiments show RLDF successfully identifies potential biases in public content while exposing limitations in human-labeled data. Our framework supports measurable progress tracking and explainable remediation actions, offering a scalable approach for improving content neutrality through transparent, multi-perspective analysis.