More Human, More Efficient: Aligning Annotations with Quantized SLMs
This provides an open-source, reproducible alternative to proprietary LLMs for text annotation, addressing biases and privacy concerns in a domain-specific context.
The paper tackles the problem of systematic biases in proprietary LLMs for text annotation by finetuning a quantized 1.7B-parameter small language model on limited human data, achieving a 0.23-point increase in inter-annotator agreement over the best proprietary LLM and demonstrating generalizability on an emotion classification task.
As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's $α$) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.