PRISM: Exposing and Resolving Spurious Isolation in Federated Multimodal Continual Learning
For federated multimodal continual learning, PRISM solves the problem of task interference and forgetting due to spurious expert isolation, achieving significant accuracy gains.
PRISM addresses spurious isolation in federated multimodal continual learning by maintaining per-expert gradient subspaces that remain orthogonal under federated averaging, outperforming 16 baselines on LLaVA and Qwen models with margins of +3.23 pp (CoIN-6) to +6.06 pp (CoIN-Long-10).
While current federated multimodal continual learning over mixture-of-experts low-rank adaptation (MoE-LoRA) is built on the unverified assumption that routing isolates task-specific knowledge into disjoint experts, we argue that routing operates per-sample, while forgetting accumulates across the task sequence, and gradient conflict persists within each expert even when routing is maximally polarized. Moreover, activation-subspace protection can also fail because, under parameter-efficient fine-tuning, it entangles tasks due to a dimension-counting bound, and federated averaging (FedAvg) disrupts client-side orthogonality. To address this, we propose PRISM (Per-expert Routing-projection Interference-informed Subspace Method), which maintains a per-expert gradient subspace basis whose orthogonality is preserved under FedAvg and reinterprets MoE routing as a capacity allocator. Our results show that, on LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-7B across CoIN-6 and CoIN-Long-10, PRISM outperforms sixteen the state of the art baselines in average accuracy. Compared to the best federated multimodal baseline, the performance margin increases from +3.23 pp on CoIN-6 to +6.06 pp on CoIN-Long-10.