Learning Human-like Representations to Enable Learning Human Values
This work addresses the challenge of personalizing AI to individual human values without causing harm, though it is incremental as it builds on known benefits of representational alignment.
The paper tackled the problem of enabling AI systems to learn human values safely and quickly by exploring representational alignment between humans and AI agents, demonstrating that this alignment improves generalization and safe exploration across ten aspects of human values, such as ethics and fairness, in a multi-armed bandit setting.
How can we build AI systems that can learn any set of individual human values both quickly and safely, avoiding causing harm or violating societal standards for acceptable behavior during the learning process? We explore the effects of representational alignment between humans and AI agents on learning human values. Making AI systems learn human-like representations of the world has many known benefits, including improving generalization, robustness to domain shifts, and few-shot learning performance. We demonstrate that this kind of representational alignment can also support safely learning and exploring human values in the context of personalization. We begin with a theoretical prediction, show that it applies to learning human morality judgments, then show that our results generalize to ten different aspects of human values -- including ethics, honesty, and fairness -- training AI agents on each set of values in a multi-armed bandit setting, where rewards reflect human value judgments over the chosen action. Using a set of textual action descriptions, we collect value judgments from humans, as well as similarity judgments from both humans and multiple language models, and demonstrate that representational alignment enables both safe exploration and improved generalization when learning human values.