Embodied Explainability and Ontological Obstacles: Why We Struggle to Explain the Answers of Large Language Models (LLMs)
For XAI researchers and practitioners, the paper challenges the dominant paradigm of extracting explanations from model internals, arguing it is ontologically flawed and can increase overreliance.
The paper argues that explainability is not a property of AI models but is created in use through embodied interaction, revealing ontological obstacles in LLM explanations that can lead to overreliance. It proposes that explainability claims should be reserved for designs providing affordances for probing, coordinating, and repairing behavior in practice.
Explainability is often framed as a property of an AI model, with explanations extracted from its internals and shown to users. In this argument paper, we instead provide an embodied account of explainability based on Dourish and enactivist cognition: understanding is created in use as people act on affordances in shared practice. Using demonstrations and conceptual analysis, we reveal ontological obstacles when "looking inside" large language models: surrogates import external abstractions that can be mistaken for the model's, and focusing on internal reasoning misses that explainers participate in their own understanding. We discuss these obstacles in XAI practice, arguing that many explanations are misnamed, which skews their purpose and can increase overreliance. Finally, we highlight how embodied explanations reorganize sense-making by making what matters publicly available for action, and argue that explainability claims should be reserved for designs that provide affordances to probe, coordinate, and repair behaviour in situated practice.