Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
This addresses efficiency challenges for deploying large vision models, but it is incremental as it builds on existing ideas of sparse activation patterns in pretrained networks.
The paper tackles the high computational demands of Vision Transformers by proposing a method to extract Mixture-of-Experts variants from pretrained models without costly retraining, achieving up to 36% reduction in MACs and 32% reduction in model size while retaining 98% of original performance on ImageNet-1k with minimal fine-tuning.
Vision Transformers have emerged as the state-of-the-art models in various Computer Vision tasks, but their high computational and resource demands pose significant challenges. While Mixture-of-Experts (MoE) can make these models more efficient, they often require costly retraining or even training from scratch. Recent developments aim to reduce these computational costs by leveraging pretrained networks. These have been shown to produce sparse activation patterns in the Multi-Layer Perceptrons (MLPs) of the encoder blocks, allowing for conditional activation of only relevant subnetworks for each sample. Building on this idea, we propose a new method to construct MoE variants from pretrained models. Our approach extracts expert subnetworks from the model's MLP layers post-training in two phases. First, we cluster output activations to identify distinct activation patterns. In the second phase, we use these clusters to extract the corresponding subnetworks responsible for producing them. On ImageNet-1k recognition tasks, we demonstrate that these extracted experts can perform surprisingly well out of the box and require only minimal fine-tuning to regain 98% of the original performance, all while reducing MACs and model size, by up to 36% and 32% respectively.