GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models
This work addresses a domain-specific problem in NLP for researchers and practitioners, offering an incremental improvement in model fine-tuning techniques.
The paper tackled the efficiency-capability trade-off in fine-tuning large language models for text classification by proposing the GTA framework, which combines supervised and reinforcement learning to achieve faster convergence and higher performance than standalone methods on four benchmarks.
In natural language processing tasks, pure reinforcement learning (RL) fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to RL. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on four text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.