Beyond Perplexity: A Lightweight Benchmark for Knowledge Retention in Supervised Fine-Tuning
This work addresses the need for better interpretability in fine-tuning for researchers and practitioners, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of distinguishing factual learning from stylistic mimicry in supervised fine-tuning of LLMs by introducing the Knowledge Retention (KR) Test, a lightweight evaluation framework that uses contrastive examples to measure likelihood preferences, validated through baseline analysis and applied to analyze LoRA training dynamics.
Supervised Fine-Tuning (SFT) is a standard approach for injecting domain knowledge into Large Language Models (LLMs). However, relying on validation perplexity to monitor training is often insufficient, as it confounds stylistic mimicry with genuine factual internalization. To address this, we introduce the Knowledge Retention (KR) Test , a lightweight, corpus-grounded evaluation framework designed to distinguish factual learning from linguistics. KR-Test utilizes automatically generated contrastive examples to measure likelihood preferences for correct versus incorrect continuations, requiring no instruction tuning or generative decoding. We validate the framework's integrity through a "blind vs. oracle" baseline analysis. Furthermore, we demonstrate the diagnostic capabilities of KR-Test by analyzing the training dynamics of Low-Rank Adaptation (LoRA). By exposing the fine-grained dissociation between linguistic convergence and knowledge retention, KR-Test enhances the interpretability of fine-tuning dynamics.