Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecture
This work addresses challenges in self-supervised learning for universal image segmentation, offering an incremental improvement with architecture-agnostic versatility.
The paper tackles the problem of training segmentation models by introducing Mask-JEPA, a self-supervised learning framework that combines Joint Embedding Predictive Architecture with mask classification architectures to capture semantics and object boundaries, achieving competitive results on datasets like ADE20K, Cityscapes, and COCO.
In this work, we introduce Mask-JEPA, a self-supervised learning framework tailored for mask classification architectures (MCA), to overcome the traditional constraints associated with training segmentation models. Mask-JEPA combines a Joint Embedding Predictive Architecture with MCA to adeptly capture intricate semantics and precise object boundaries. Our approach addresses two critical challenges in self-supervised learning: 1) extracting comprehensive representations for universal image segmentation from a pixel decoder, and 2) effectively training the transformer decoder. The use of the transformer decoder as a predictor within the JEPA framework allows proficient training in universal image segmentation tasks. Through rigorous evaluations on datasets such as ADE20K, Cityscapes and COCO, Mask-JEPA demonstrates not only competitive results but also exceptional adaptability and robustness across various training scenarios. The architecture-agnostic nature of Mask-JEPA further underscores its versatility, allowing seamless adaptation to various mask classification family.