Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
This work addresses a practical problem for practitioners adapting language models to new tasks, but it is incremental as it builds on prior synthetic data and replay techniques.
The study tackled the trade-off between learning new tasks and avoiding catastrophic forgetting in language models by investigating optimal replay ratios for balancing task performance and knowledge retention under computational constraints, finding an optimal configuration that reduces training costs.
Adapting language models to new tasks through continued pretraining faces a fundamental trade-off: models must learn new capabilities while avoiding catastrophic forgetting of existing knowledge. While prior work has studied synthetic data generation techniques, the optimal replay ratios for balancing task performance and knowledge retention under computational constraints remain poorly understood. We present a comprehensive empirical study investigating the interplay between replay ratio configuration and computational budget when adapting language models to new tasks. Using the bAbI reasoning tasks as our target objective, we apply synthetic data generation and systematically evaluate different total token budgets and replay ratio configurations. We analyze their effects on both task mastery and general knowledge retention. Our experiments reveal an optimal configuration that balances task-specific performance with general knowledge retention. Based on our findings, we provide empirically-grounded guidelines for selecting replay ratios based on computational budget, enabling practitioners to achieve strong task adaptation with significantly reduced training costs.