SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs
For LLM alignment in human-centered tasks with label ambiguity, SHALA-LLM provides a method to model disagreement as information rather than noise, improving both distributional agreement and classification performance.
SHALA-LLM introduces a reinforcement learning framework that enables LLMs to learn from annotator distributions and prioritize ambiguous samples, improving agreement with human label distributions (e.g., 62.1% reduction in Jensen-Shannon Distance on ChaosNLI) and boosting F1 by up to 16.7% on NLI and emotion recognition benchmarks.
Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challenging disagreements across human annotators. As LLMs are increasingly deployed in real-world settings, faithfully modeling such ambiguity is essential to identify contested inputs, preserve variability in ambiguous cases, and capture the full distribution of human judgments. Yet, existing LLM alignment approaches have predominantly assumed a single correct label, excluding annotator disagreement during optimization. Instead of treating this ambiguity as noise, we show how to treat it as information that improves model behavior through a new algorithm called SMARTLY HANDLING AMBIGUOUS LABELS IN ALIGNING LLMS (SHALA-LLM). This reinforcement learning framework provides a new way for LLMs to learn directly from annotator distributions while dynamically prioritizing highly ambiguous samples during optimization. Experiments on ambiguity-sensitive NLI and ER benchmarks, including ChaosNLI, GoEmotions, and MSP-Podcast, demonstrate that SHALA-LLM improves agreement with annotator label distributions, e.g. on ChaosNLI, it reduces Jensen-Shannon Distance by up to 62.1%. At the same time, SHALA-LLM improves F1 by up to 16.7%, showing that modeling annotator disagreement can also strengthen classification performance.