YOLOR-Based Multi-Task Learning
This work addresses the problem of multi-task learning for computer vision researchers, but it is incremental as it extends the YOLOR architecture to more tasks.
The paper tackles the challenge of jointly training multiple vision tasks (object detection, instance segmentation, semantic segmentation, and image captioning) using a single model, achieving competitive performance across all tasks while maintaining a low parameter count and no pre-training.
Multi-task learning (MTL) aims to learn multiple tasks using a single model and jointly improve all of them assuming generalization and shared semantics. Reducing conflicts between tasks during joint learning is difficult and generally requires careful network design and extremely large models. We propose building on You Only Learn One Representation (YOLOR), a network architecture specifically designed for multitasking. YOLOR leverages both explicit and implicit knowledge, from data observations and learned latents, respectively, to improve a shared representation while minimizing the number of training parameters. However, YOLOR and its follow-up, YOLOv7, only trained two tasks at once. In this paper, we jointly train object detection, instance segmentation, semantic segmentation, and image captioning. We analyze tradeoffs and attempt to maximize sharing of semantic information. Through our architecture and training strategies, we find that our method achieves competitive performance on all tasks while maintaining a low parameter count and without any pre-training. We will release code soon.