Risk-Averse Finetuning of Large Language Models
This addresses safety concerns for users and platforms by mitigating harmful outputs in LLMs, though it is incremental as it builds on existing RLHF methods.
The paper tackled the problem of reducing toxic content generation by Large Language Models by integrating risk-averse principles into fine-tuning, resulting in improved avoidance of harmful outputs while maintaining generative task effectiveness.
We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts. We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events. By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.