Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models
This addresses the challenge of safely deploying language models in real-world applications by controlling toxicity, representing a novel method for a known bottleneck.
The paper tackles the problem of reducing toxic content generation in pretrained large language models by proposing two pretraining data augmentation strategies, MEDA and INST, with INST reducing toxicity probability by up to 61% while maintaining accuracy on NLP tasks and improving bias detection AUC scores by 1.3%.
Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks. However, safely deploying them in real world applications is challenging because they generate toxic content. To address this challenge, we propose two novel pretraining data augmentation strategies that significantly reduce model toxicity without compromising its utility. Our two strategies are: (1) MEDA: adds raw toxicity score as meta-data to the pretraining samples, and (2) INST: adds instructions to those samples indicating their toxicity. Our results indicate that our best performing strategy (INST) substantially reduces the toxicity probability up to 61% while preserving the accuracy on five benchmark NLP tasks as well as improving AUC scores on four bias detection tasks by 1.3%. We also demonstrate the generalizability of our techniques by scaling the number of training samples and the number of model parameters.