From Particles to Agents: Hallucination as a Metric for Cognitive Friction in Spatial Simulation
This work addresses a problem for architectural simulation and HCI researchers by proposing a novel framework that treats environments as dynamic cognitive partners, though it appears incremental as it builds on existing multimodal and simulation concepts.
The paper tackles the limitation of traditional architectural simulations that model elements as deterministic physics-based particles rather than cognitive agents by introducing Agentic Environmental Simulations, which use Large Multimodal generative models to predict spatial states based on semantic expectation and shift to Episodic Spatial Reasoning through surprisal-triggered events, with results including the formalization of Cognitive Friction (C_f) to reveal Phantom Affordances as semiotic ambiguities in built space.
Traditional architectural simulations (e.g. Computational Fluid Dynamics, evacuation, structural analysis) model elements as deterministic physics-based "particles" rather than cognitive "agents". To bridge this, we introduce \textbf{Agentic Environmental Simulations}, where Large Multimodal generative models actively predict the next state of spatial environments based on semantic expectation. Drawing on examples from accessibility-oriented AR pipelines and multimodal digital twins, we propose a shift from chronological time-steps to Episodic Spatial Reasoning, where simulations advance through meaningful, surprisal-triggered events. Within this framework we posit AI hallucinations as diagnostic tools. By formalizing the \textbf{Cognitive Friction} ($C_f$) it is possible to reveal "Phantom Affordances", i.e. semiotic ambiguities in built space. Finally, we challenge current HCI paradigms by treating environments as dynamic cognitive partners and propose a human-centered framework of cognitive orchestration for designing AI-driven simulations that preserve autonomy, affective clarity, and cognitive integrity.