Pathformer3D: A 3D Scanpath Transformer for 360° Images
This work addresses a domain-specific problem for Virtual/Augmented Reality applications by improving scanpath prediction accuracy, though it appears incremental as it builds on existing Transformer-based methods.
The paper tackles scanpath prediction in 360° images by proposing Pathformer3D, a 3D scanpath Transformer that operates in a spherical coordinate system to reduce errors from 2D distortions, and it outperforms state-of-the-art methods on four panoramic eye-tracking datasets.
Scanpath prediction in 360° images can help realize rapid rendering and better user interaction in Virtual/Augmented Reality applications. However, existing scanpath prediction models for 360° images execute scanpath prediction on 2D equirectangular projection plane, which always result in big computation error owing to the 2D plane's distortion and coordinate discontinuity. In this work, we perform scanpath prediction for 360° images in 3D spherical coordinate system and proposed a novel 3D scanpath Transformer named Pathformer3D. Specifically, a 3D Transformer encoder is first used to extract 3D contextual feature representation for the 360° image. Then, the contextual feature representation and historical fixation information are input into a Transformer decoder to output current time step's fixation embedding, where the self-attention module is used to imitate the visual working memory mechanism of human visual system and directly model the time dependencies among the fixations. Finally, a 3D Gaussian distribution is learned from each fixation embedding, from which the fixation position can be sampled. Evaluation on four panoramic eye-tracking datasets demonstrates that Pathformer3D outperforms the current state-of-the-art methods. Code is available at https://github.com/lsztzp/Pathformer3D .