Task-Oriented Communications for 3D Scene Representation: Balancing Timeliness and Fidelity
This work addresses a critical problem for applications like VR/AR/MR and the metaverse by optimizing the trade-off in dynamic environments, though it is incremental as it builds on existing methods like PPO and AoI.
The paper tackles the challenge of balancing timeliness and fidelity in real-time 3D scene representation by proposing a contextual-bandit PPO framework that optimizes image selection based on Age of Information and semantic information, demonstrating improved representation fidelity with low latency in experiments.
Real-time Three-dimensional (3D) scene representation is a foundational element that supports a broad spectrum of cutting-edge applications, including digital manufacturing, Virtual, Augmented, and Mixed Reality (VR/AR/MR), and the emerging metaverse. Despite advancements in real-time communication and computing, achieving a balance between timeliness and fidelity in 3D scene representation remains a challenge. This work investigates a wireless network where multiple homogeneous mobile robots, equipped with cameras, capture an environment and transmit images to an edge server over channels for 3D representation. We propose a contextual-bandit Proximal Policy Optimization (PPO) framework incorporating both Age of Information (AoI) and semantic information to optimize image selection for representation, balancing data freshness and representation quality. Two policies -- the $ω$-threshold and $ω$-wait policies -- together with two benchmark methods are evaluated, timeliness embedding and weighted sum, on standard datasets and baseline 3D scene representation models. Experimental results demonstrate improved representation fidelity while maintaining low latency, offering insight into the model's decision-making process. This work advances real-time 3D scene representation by optimizing the trade-off between timeliness and fidelity in dynamic environments.