Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models
This work addresses a specific bottleneck in training language models with reinforcement learning, offering an incremental improvement for AI safety and alignment.
The paper tackles the problem of sub-optimal text generation in reinforcement learning for large language models by proposing a utility-inspired reward transformation, showing that it improves helpfulness and reduces harm compared to linear aggregation methods.
Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This overlooks crucial aspects of individual reward dimensions and inter-reward dependencies that can lead to sub-optimal outcomes in generations. In this work, we show how linear aggregation of rewards exhibits some vulnerabilities that can lead to undesired properties of generated text. We then propose a transformation of reward functions inspired by economic theory of utility functions (specifically Inada conditions), that enhances sensitivity to low reward values while diminishing sensitivity to already high values. We compare our approach to the existing baseline methods that linearly aggregate rewards and show how the Inada-inspired reward feedback is superior to traditional weighted averaging. We quantitatively and qualitatively analyse the difference in the methods, and see that models trained with Inada-transformations score as more helpful while being less harmful.