Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
For researchers developing continual learning benchmarks for MLLMs, this paper exposes structural flaws in the current MLLM-CL benchmark and proposes design principles for future benchmarks.
The paper re-evaluates the MR-LoRA method for continual learning in MLLMs, finding that routing does not require an MLLM (a simple prototype method matches performance at lower cost) and shared experts do not help. These results stem from benchmark limitations: tasks are highly separable and fixed order makes conclusions sensitive to curriculum, rewarding isolated learning over transfer.
Continual adaptation is essential for multimodal large language models (MLLMs) deployed across evolving domains, but the state-of-the-art MR-LoRA method highly relies on the assumption that a MLLM-based router is necessary to process complex multimodal inputs. This paper revisits this claim on the MLLM-CL benchmark and argues for two claims. \textbf{First}, routing does not require an MLLM: a simple training-free, replay-free ptotypical routing method (\textsc{RePRo}), uses frozen pretrained features and task prototypes to match the MLLM-based router of MR-LoRA at far lower computational cost. \textbf{Second}, shared experts do not improve continual learning for MLLMs, despite their theoretical appeal. We show that these findings arise from two structural limitations of MLLM-CL: (1) its tasks are \textbf{highly separable} in representation space, and (2) its fixed task order makes conclusions \textbf{sensitive to a single curriculum} rather than robust across diverse continual-learning trajectories. As a result, the benchmark primarily rewards learning in isolation rather than genuine continual transfer. This motivates a new design for future benchmarks of continual MLLM learning, with overlapping task manifolds, multiple task orders, fine-grained domain shifts, and evaluation protocols that reward forward transfer as well as retention.