Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models
This work addresses the need for comprehensive fairness evaluation in SLMs for resource-constrained applications, though it is incremental as it builds on existing bias assessment methods.
The authors tackled the problem of assessing fairness in Small Language Models (SLMs) by introducing the Vacuous Neutrality Framework (VaNeu), a multi-dimensional evaluation paradigm, and conducted the first large-scale audit of nine SLMs in the 0.5-5B parameter range, revealing that models with low bias in early stages often fail later evaluations, uncovering hidden vulnerabilities.
The rapid adoption of Small Language Models (SLMs) for resource constrained applications has outpaced our understanding of their ethical and fairness implications. To address this gap, we introduce the Vacuous Neutrality Framework (VaNeu), a multi-dimensional evaluation paradigm designed to assess SLM fairness prior to deployment. The framework examines model robustness across four stages - biases, utility, ambiguity handling, and positional bias over diverse social bias categories. To the best of our knowledge, this work presents the first large-scale audit of SLMs in the 0.5-5B parameter range, an overlooked "middle tier" between BERT-class encoders and flagship LLMs. We evaluate nine widely used SLMs spanning four model families under both ambiguous and disambiguated contexts. Our findings show that models demonstrating low bias in early stages often fail subsequent evaluations, revealing hidden vulnerabilities and unreliable reasoning. These results underscore the need for a more comprehensive understanding of fairness and reliability in SLMs, and position the proposed framework as a principled tool for responsible deployment in socially sensitive settings.