A Taxonomy of Confabulations and the Perception-Reality Gap in LLM-Assisted Immersive Scene Editing
For researchers and designers of LLM-assisted immersive systems, this work provides a taxonomy and user insights to guide confabulation mitigation, though it is an exploratory study with limited scope.
This paper studies LLM confabulations in immersive 3D scene editing, constructing a taxonomy from a study with 24 non-expert users. It reports prevalence and disruptiveness of confabulations and defines the perception-reality gap, finding that users' awareness of confabulations saturates under cognitive load.
Large language models (LLMs) are being increasingly integrated into immersive environments and design workflows, providing application prospects in areas such as rapid scene prototyping for non-expert users and scene understanding capabilities for accessibility design. While many workflows that incorporate LLMs in immersive spaces are proposed, such systems can exhibit errors, potentially resulting in frustration, loss of user trust, and compromised user safety. This paper studies the underexplored area of LLM confabulations in immersive 3D scene editing contexts. Through an exploratory study with 24 non-expert users, we construct a taxonomy of the different types of confabulation observed in LLM-assisted immersive 3D scene editing. We report their prevalence and disruptiveness, and define the construct perception-reality gap to help understand the gap between the actual and perceived occurrence of confabulations. We highlight the observed saturation of confabulation awareness under load and conclude by discussing design implications for confabulation mitigation in future LLM-assisted systems in immersive 3D scenes.