Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
This addresses the challenge of multi-style controllable text generation for applications like content creation, though it is incremental as it builds on existing multi-objective RL methods.
The paper tackled the problem of controlling language models to generate text combining multiple target styles, such as negative and non-toxic, by investigating multi-objective reinforcement learning reward formulations. It found that dynamic weighting based on discriminator gradient magnitudes outperforms static approaches in style control while maintaining linguistic quality, with results explored for 2- and 3-style control.
Textual style expresses a diverse set of information, including interpersonal dynamics (e.g., formality) and the author's emotions or attitudes (e.g., disgust). An open question is how language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. One approach to such controlled generation is multi-objective reinforcement learning (RL), but how best to combine multiple objectives in a reward function is an open question. In this paper, we investigate various formulations of multi-style rewards, including calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that our proposed dynamic weighting outperforms static weighting approaches with respect to style control while maintaining linguistic quality, and we explore its effectiveness in 2- and 3-style control.