Align AI to Dynamic Human-AI Workflows
For AI alignment researchers, this paper identifies a conceptual gap in existing approaches and proposes a new direction, but it is primarily a position paper without empirical results.
The paper argues that current AI alignment methods fail to capture the dynamic, context-dependent nature of real-world human-AI interactions, proposing a shift to interactive and complementary alignment where preferences emerge through interaction. It outlines a research agenda for developing AI systems that align with humans in interaction, drawing on interdisciplinary insights.
Current alignment approaches typically focus on emulating human behavior using static representations of human preferences, failing to capture the dynamic, context-dependent nature of real-world human-AI interactions. In this paper, we argue for a shift from static and emulative to interactive and complementary alignment, where preferences emerge through interaction and alignment is defined not by satisfying preferences alone. We first formalize this gap by contrasting existing alignment with a trajectory-level view in which human and model behavior co-evolve over time. Because these interaction dynamics have not been adequately captured within existing ML formulations, we ground this perspective in insights from an interdisciplinary workshop. We draw on lessons from social-science accounts of human-human collaboration and then argue that human-AI systems amplify these dynamics, introducing new asymmetries that make reasoning about uncertainty harder and introduce new coordination challenges. Based on these lessons and new challenges, we conclude by outlining a research agenda for developing AI systems that align with humans in interaction, requiring an interdisciplinary synthesis of machine learning and the social and decision sciences.