Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
This work addresses the problem of toxic language generation in large language models for AI safety and deployment, presenting incremental improvements in detoxification methods.
The paper systematically explores domain-adaptive training to reduce toxicity in large-scale language models, finding that self-generated nontoxic datasets outperform baselines with a 1/3 smaller corpus and that large models require more effort to detoxify while adapter layers offer a better trade-off between toxicity and perplexity.
Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.