(Almost) Free Modality Stitching of Foundation Models
This addresses a critical bottleneck in efficiently building multi-modal AI systems, offering a significant computational saving for researchers and practitioners.
The paper tackles the computationally demanding problem of selecting and training connector modules for stitching foundation models by proposing Hypernetwork Model Alignment (Hyma), which reduces the cost of searching for the best uni-modal model pair by 10× while matching performance on multi-modal benchmarks.
Foundation multi-modal models are often designed by stitching of multiple existing pretrained uni-modal models: for example, an image classifier with an text model. This stitching process is performed by training a connector module that aims to align the representation spaces of these uni-modal models towards a multi-modal objective. However, given the complexity of training such connectors on large scale web-based datasets coupled with the ever-increasing number of available pretrained uni-modal models, the task of uni-modal models selection and subsequent connector module training becomes computationally demanding. To address this under-studied critical problem, we propose Hypernetwork Model Alignment (Hyma), a novel all-in-one solution for optimal uni-modal model selection and connector training by leveraging hypernetworks. Specifically, our framework utilizes the parameter prediction capability of a hypernetwork to obtain jointly trained connector modules for $N \times M$ combinations of uni-modal models. In our experiments, Hyma reduces the cost of searching for the best performing uni-modal model pair by $10\times$, while matching the ranking and trained connector performance obtained via grid search across a suite of diverse multi-modal benchmarks.