Self-Generative Adversarial Fine-Tuning for Large Language Models
This addresses the challenge of reducing dependence on expensive human feedback for AI alignment, offering a novel method that could benefit developers and researchers in natural language processing.
The paper tackles the problem of fine-tuning large language models for alignment without costly human annotations by proposing Self-Generative Adversarial LLM (SGALM), a framework that formulates alignment as a generative adversarial game within a single model, achieving state-of-the-art performance.
Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.