Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
This addresses the problem of accurate future semantic segmentation for autonomous navigation, representing an incremental improvement with novel components.
The paper tackles semantic future prediction for autonomous systems by introducing FUTURIST, a multimodal visual sequence transformer method that achieves state-of-the-art performance on the Cityscapes dataset for short- and mid-term forecasting.
Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. We provide the implementation code at https://github.com/Sta8is/FUTURIST .