Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
This addresses alignment issues in language models for users seeking safer and more reliable AI outputs, though it is incremental as it builds on existing RL methods.
The paper tackled the problem of reducing undesirable outputs in language models while maintaining average-case performance, introducing RePULSe, a training method that improved the tradeoff between expected reward and probability of undesired outputs compared to standard RL approaches.
Reinforcement learning (RL) has become a predominant technique to align language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL alignment approaches and alternatives.