Scaling Law Hypothesis for Multimodal Model
This work addresses the challenge of efficiently deploying multimodal models on resource-constrained devices, but it appears incremental as it extends existing scaling laws to new modalities.
The authors tackled the problem of predicting performance for multimodal models by proposing a scaling law hypothesis that extends text-based scaling laws to mixed-modality systems, based on modality-specific compression and tokenization efficiency.
We propose a scaling law hypothesis for multimodal models processing text, audio, images, and video within a shared token and embedding space. Our framework predicts model performance based on modality-specific compression and tokenization efficiency, extending established scaling laws from text-based decoder models to mixed-modality systems. We explore whether leveraging more training data in multiple modalities can reduce the size of the multimodal model, enabling efficient deployment on resource-constrained devices.