A Human Digital Twin Architecture for Knowledge-based Interactions and Context-Aware Conversations
This addresses the problem of building trust and shared contextual understanding in Human-Autonomy Teaming for tasks and missions, though it appears incremental as it builds on existing AI and ML developments.
The paper tackles the challenge of enabling humans to maintain awareness and control over autonomous assets in Human-Autonomy Teaming by presenting a real-time Human Digital Twin architecture that integrates Large Language Models for knowledge-based interactions and context-aware conversations, resulting in a system with performance metrics for adaptive and realistic teaming.
Recent developments in Artificial Intelligence (AI) and Machine Learning (ML) are creating new opportunities for Human-Autonomy Teaming (HAT) in tasks, missions, and continuous coordinated activities. A major challenge is enabling humans to maintain awareness and control over autonomous assets, while also building trust and supporting shared contextual understanding. To address this, we present a real-time Human Digital Twin (HDT) architecture that integrates Large Language Models (LLMs) for knowledge reporting, answering, and recommendation, embodied in a visual interface. The system applies a metacognitive approach to enable personalized, context-aware responses aligned with the human teammate's expectations. The HDT acts as a visually and behaviorally realistic team member, integrated throughout the mission lifecycle, from training to deployment to after-action review. Our architecture includes speech recognition, context processing, AI-driven dialogue, emotion modeling, lip-syncing, and multimodal feedback. We describe the system design, performance metrics, and future development directions for more adaptive and realistic HAT systems.