$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
This addresses the problem of domain shift in activation-aware compression for unknown downstream tasks, offering an incremental improvement in efficient inference for large models.
The paper tackles the computational demand of large foundation models by introducing a test-time pruning method that adapts to each prompt without retraining, achieving reduced inference complexity through dynamic structured sparsity.
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called $μ$-MoE. Several experiments demonstrate that $μ$-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.