Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs
This work addresses the need for better tools to analyze and adjust refusal behaviors in large language models, which is crucial for AI safety and reliability, though it is incremental in building on existing IFT/RLHF datasets.
The paper tackled the problem of inadequate taxonomies and evaluation datasets for LLM refusals by developing a comprehensive framework with a 16-category taxonomy, human-annotated dataset of over 8,600 instances, synthetic dataset with 8,000 examples per category, and classifiers for refusal classification, enabling precise auditing and analysis of refusal behaviors in black-box LLMs.
Refusals - instances where large language models (LLMs) decline or fail to fully execute user instructions - are crucial for both AI safety and AI capabilities and the reduction of hallucinations in particular. These behaviors are learned during post-training, especially in instruction fine-tuning (IFT) and reinforcement learning from human feedback (RLHF). However, existing taxonomies and evaluation datasets for refusals are inadequate, often focusing solely on should-not-related (instead of cannot-related) categories, and lacking tools for auditing refusal content in black-box LLM outputs. We present a comprehensive framework for classifying LLM refusals: (a) a taxonomy of 16 refusal categories, (b) a human-annotated dataset of over 8,600 instances from publicly available IFT and RLHF datasets, (c) a synthetic dataset with 8,000 examples for each refusal category, and (d) classifiers trained for refusal classification. Our work enables precise auditing of refusal behaviors in black-box LLMs and automatic analyses of refusal patterns in large IFT and RLHF datasets. This facilitates the strategic adjustment of LLM refusals, contributing to the development of more safe and reliable LLMs.