Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models
This addresses inefficiency in debiasing for LLMs, offering a more practical solution for real-world applications, though it is incremental as it builds on existing distillation and debiasing techniques.
The paper tackles the problem of high computational cost at inference time for debiasing large language models (LLMs) against permutation sensitivity, by distilling a debiased teacher model into a compact student model, achieving better results with significantly fewer parameters.
Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they sometimes fail to maintain crucial invariances for specific tasks. One example is permutation sensitivity, where LLMs' outputs may significantly vary depending on the order of the input options. While debiasing techniques can mitigate these issues, and yield better performance and reliability, they often come with a high computational cost at inference. This paper addresses this inefficiency at inference time. The aim is to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model. We explore two variants of student models: one based on pure distillation, and the other on an error-correction approach for more complex tasks, where the student corrects a single biased decision from the teacher to achieve a debiased output. Our approach is general and can be applied to both black-box and white-box LLMs. Furthermore, we demonstrate that our compact, encoder-only student models can outperform their larger, biased teacher counterparts, achieving better results with significantly fewer parameters.