GNN-MoE: Context-Aware Patch Routing using GNNs for Parameter-Efficient Domain Generalization
This addresses domain generalization for vision tasks, offering a lightweight solution, but it is incremental as it builds on existing PEFT and MoE frameworks.
The paper tackled the problem of domain generalization for Vision Transformers by proposing GNN-MoE, a parameter-efficient fine-tuning method that uses a GNN router to assign patches to experts based on inter-patch relationships, achieving state-of-the-art or competitive performance on benchmarks with high parameter efficiency.
Domain generalization (DG) seeks robust Vision Transformer (ViT) performance on unseen domains. Efficiently adapting pretrained ViTs for DG is challenging; standard fine-tuning is costly and can impair generalization. We propose GNN-MoE, enhancing Parameter-Efficient Fine-Tuning (PEFT) for DG with a Mixture-of-Experts (MoE) framework using efficient Kronecker adapters. Instead of token-based routing, a novel Graph Neural Network (GNN) router (GCN, GAT, SAGE) operates on inter-patch graphs to dynamically assign patches to specialized experts. This context-aware GNN routing leverages inter-patch relationships for better adaptation to domain shifts. GNN-MoE achieves state-of-the-art or competitive DG benchmark performance with high parameter efficiency, highlighting the utility of graph-based contextual routing for robust, lightweight DG.