MulT: An End-to-End Multitask Learning Transformer
This work addresses the challenge of efficient multitask learning in computer vision, offering a robust solution that generalizes across domains, though it is incremental as it builds on existing transformer architectures.
The authors tackled the problem of simultaneously learning multiple high-level vision tasks by proposing MulT, an end-to-end multitask learning transformer framework, which outperformed state-of-the-art multitask CNN models and single-task transformer models on several benchmarks.
We propose an end-to-end Multitask Learning Transformer framework, named MulT, to simultaneously learn multiple high-level vision tasks, including depth estimation, semantic segmentation, reshading, surface normal estimation, 2D keypoint detection, and edge detection. Based on the Swin transformer model, our framework encodes the input image into a shared representation and makes predictions for each vision task using task-specific transformer-based decoder heads. At the heart of our approach is a shared attention mechanism modeling the dependencies across the tasks. We evaluate our model on several multitask benchmarks, showing that our MulT framework outperforms both the state-of-the art multitask convolutional neural network models and all the respective single task transformer models. Our experiments further highlight the benefits of sharing attention across all the tasks, and demonstrate that our MulT model is robust and generalizes well to new domains. Our project website is at https://ivrl.github.io/MulT/.