The End of Human Judgment in the Kill Chain? Relocating Initiative and Interpretation with Agentic AI
This addresses the problem of ensuring human oversight in military AI systems for policymakers and governance bodies, highlighting a critical ethical and operational challenge.
The paper argues that LLM-based agents in battlefield functions, such as data fusion and battle management, displace human decision-making, making them incompatible with existing governance frameworks requiring human judgment and control. It concludes that these applications cannot be used justifiably in lethal contexts under current conditions and proposes responses for international governance.
Large language model-based agents are increasingly being integrated into core battlefield functions, including intelligence analysis, data fusion, and battlefield management. This paper argues that the very features that make such agents operationally attractive, namely their capacity for initiative, interpretation, their goal-directedness, and dynamic memory, are the same features that render context-appropriate human judgment and control substantively ineffectual in those parts of the kill chain where agents operate. Drawing on specific use cases, the paper argues that by relocating initiative and interpretation, LLM-based agents displace human decision-making in ways that makes their use incompatible with the requirement of human judgment and control which is central to existing governance frameworks, like those proposed by the GGE-CCW and REAIM. The paper concludes that a subset of agentic AI applications, particularly those deployed for data fusion and battle management in lethal contexts, cannot be used justifiably on the battlefield under current and foreseeable conditions, and proposes two ways for the international governance community to respond to this challenge.