SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation
This addresses fairness issues in LLMs for responsible AI deployment, representing a novel application of dynamic validation in in-context learning.
The study tackled the problem of ensuring fairness in LLMs for tasks like tabular classification by introducing a dynamic validation set and an iterative algorithm (SMITE) to select optimal in-context examples, resulting in significant improvements in both predictive accuracy and fairness across four LLMs compared to baselines.
Large Language Models (LLMs) are widely used for downstream tasks such as tabular classification, where ensuring fairness in their outputs is critical for inclusivity, equal representation, and responsible AI deployment. This study introduces a novel approach to enhancing LLM performance and fairness through the concept of a dynamic validation set, which evolves alongside the test set, replacing the traditional static validation approach. We also propose an iterative algorithm, SMITE, to select optimal in-context examples, with each example set validated against its corresponding dynamic validation set. The in-context set with the lowest total error is used as the final demonstration set. Our experiments across four different LLMs show that our proposed techniques significantly improve both predictive accuracy and fairness compared to baseline methods. To our knowledge, this is the first study to apply dynamic validation in the context of in-context learning for LLMs.