AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
For practitioners of continual multimodal learning, AlphaWiSE offers a post-hoc method to balance stability and plasticity without additional inference cost.
AlphaWiSE addresses the problem of cross-modal alignment disruption during continual adaptation of multimodal models like CLIP. It achieves consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics on audio-image-text retrieval.
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.