ES-Merging: Biological MLLM Merging via Embedding Space Signals
This work addresses the limitation of specialized MLLMs in handling cross-modal scientific problems, offering an efficient merging solution for researchers in computational biology and AI.
The paper tackles the problem of merging specialized multimodal large language models (MLLMs) for cross-modal scientific tasks by proposing a representation-aware framework that estimates merging coefficients from embedding space signals, achieving performance that outperforms existing merging methods and even surpasses task-specific fine-tuned models on interactive effect prediction benchmarks.
Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging.