LLM-Mediated Human-AI Interaction in Search and Rescue: Impact of Expertise on Attentional Allocation
For human-AI teaming researchers, this work reveals that LLM-mediated assistance creates an attention-guidance trade-off moderated by expertise, highlighting the need for designs that support situational awareness.
This study investigates how LLM-based guidance affects human performance and attention in search and rescue tasks, finding that while it improves efficiency (higher rewards and victims-per-step), it does not increase total victims saved and shifts visual attention to the chat interface, with novices showing passive reliance and experts maintaining environmental scanning.
Human-AI teaming (HAT) increasingly involves AI systems that provide real-time, context-aware guidance in complex tasks. While such systems can improve performance, their effectiveness depends on how they shape human cognition and behavior. In particular, AI assistance can introduce cognitive demands and influence attention, planning, and interaction with the task environment, with effects that can vary across levels of expertise. This work investigates these mechanisms in a simulated search and rescue (SAR) environment. We compare human performance under two LLM (Large Language Model)-guided conditions and a no-LLM baseline, and analyze interaction at multiple levels, including task performance, eye-tracking measures, and planning behavior. Eye tracking provides fine-grained insight into attention allocation and interaction with AI guidance, while behavioral measures capture how users structure and adapt their decisions over time. Results indicate that LLM guidance enhanced task efficiency (higher rewards and victims-per-step) but did not increase total victims saved. Eye-tracking data revealed an attention-guidance trade-off, with visual resources shifting to the chat interface alongside increased pupil size variability. Expertise moderated this effect: novices exhibited passive AI reliance, whereas experts maintained a "verification loop" through persistent environmental scanning. These findings suggest that LLM-mediated teaming efficacy depends on the operator's ability to cross-reference AI guidance with ground truth to maintain situational awareness.