Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
This addresses the issue of value misalignment in AI systems for users and developers, with incremental improvements in interpretability and error correction.
The paper tackles the problem of aligning language models with human values by modeling text edits, achieving superior performance on three benchmark datasets and demonstrating strong transfer learning in few-shot scenarios.
We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thought not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.