Diversity in Large Language Models under Supervised Fine-Tuning
For practitioners fine-tuning LLMs, this work addresses the overlooked issue of diversity loss, offering a principled solution to improve output variety without sacrificing quality.
The study identifies that supervised fine-tuning reduces generative diversity in LLMs due to neglect of low-frequency patterns and forgetting of preexisting knowledge, and proposes TOFU loss to enhance diversity while maintaining response quality.
Supervised Fine-Tuning (SFT) is essential for aligning Large Language Models (LLMs) with user intent, yet it is believed to suppress generative diversity. Although this reduction is frequently referenced, formal empirical testing of the phenomenon remains limited. The expressiveness of LLMs by itself was addressed by multiple prior methods. Their varying perspectives suggest that deeper analysis could yield further improvements. In this study, we attribute the decline to two primary drivers: the neglect of low-frequency patterns within fine-tuning datasets and the forgetting of preexisting knowledge. Motivated by our theoretical analysis, we develop Tempered Focal (TOFU) loss, a novel objective that addresses both stated challenges simultaneously. Our extensive evaluation confirms at scale that generation breadth narrows after SFT and strengthens the hypothesis explaining this effect. Across multiple models and benchmarks, we demonstrate that TOFU enhances output diversity while preserving high response quality, offering a principled approach to SFT.