REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations
For practitioners fine-tuning LLMs with crowd-sourced annotations, REALM provides a robust method to mitigate the impact of unreliable annotators without requiring additional supervision.
REALM jointly learns model parameters and per-annotator expertise from noisy labels without supervision, improving fine-tuning accuracy by up to 50% over naive SFT across five QA benchmarks and three model sizes.
Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators directly into its parameters. We propose REALM, a method that jointly learns the model parameters and a scalar expertise value for each annotator entirely unsupervised, requiring no supervision beyond annotator identity. The key idea is to model each observed label as a mixture between the model's prediction and a uniform random guess, weighted by the annotator's learned expertise. We extend REALM to a multi-task setting via a learned expertise matrix that captures per-annotator reliability across tasks. We evaluate on five question answering benchmarks, fine-tuning three sizes of Flan-T5 under simulated noisy annotations. The proposed algorithm consistently outperforms the naive noisy SFT in the large majority of single- and multi-task settings, across datasets, model sizes, and noise types, with accuracy improvements of up to $50\%$ in the most adversarial regime and gains that grow with model capacity.