Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
This addresses the limitation of cross-entropy training for language models by providing a method to optimize sequence-level semantics, which is incremental as it builds on existing fine-tuning approaches.
The paper tackled the problem of cross-entropy training optimizing next-token prediction rather than sequence-level behavior by introducing a feature-matching objective for fine-tuning language models, resulting in EBFT outperforming SFT and matching RLVR on downstream accuracy across tasks like Q&A coding and translation.
Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level behavior under model rollouts. We introduce a feature-matching objective for language-model fine-tuning that targets sequence-level statistics of the completion distribution, providing dense semantic feedback without requiring a task-specific verifier or preference model. To optimize this objective efficiently, we propose energy-based fine-tuning (EBFT), which uses strided block-parallel sampling to generate multiple rollouts from nested prefixes concurrently, batches feature extraction over these rollouts, and uses the resulting embeddings to perform an on-policy policy-gradient update. We present a theoretical perspective connecting EBFT to KL-regularized feature-matching and energy-based modeling. Empirically, across Q&A coding, unstructured coding, and translation, EBFT matches RLVR and outperforms SFT on downstream accuracy while achieving a lower validation cross-entropy than both methods.