MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance
This work addresses the problem of limited scalability and feature extraction in LSTM-based models for computer vision researchers, representing an incremental advancement over existing xLSTM methods.
The paper tackles the challenge of scaling LSTM networks for visual tasks by introducing MAL, a framework that enhances xLSTM through cluster-masked pretraining and multi-task learning, resulting in improved performance and setting a new benchmark in visual tasks.
The Long Short-Term Memory (LSTM) networks have traditionally faced challenges in scaling and effectively capturing complex dependencies in visual tasks. The xLSTM architecture has emerged to address these limitations, incorporating exponential gating and a parallel matrix memory structure to enhance performance and scalability. Despite these advancements, the potential of xLSTM in visual computing has not been fully realized, particularly in leveraging autoregressive techniques for improved feature extraction. In this paper, we introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimizes image scanning efficiency. Additionally, our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks. Our experimental results demonstrate that MAL surpasses traditional supervised models and fully leverages the scaling potential of xLSTM, setting a new benchmark in visual task performance.