MOSEL: Inference Serving Using Dynamic Modality Selection
This addresses the problem of high computational costs in inference serving for applications requiring multi-modal models, offering an incremental improvement over existing systems.
The paper tackles the challenge of serving multi-modal ML models efficiently by introducing modality selection, which adaptively chooses input modalities per request to meet latency and cost requirements while maintaining accuracy, resulting in a 3.6x throughput improvement and 11x shorter job completion times.
Rapid advancements over the years have helped machine learning models reach previously hard-to-achieve goals, sometimes even exceeding human capabilities. However, to attain the desired accuracy, the model sizes and in turn their computational requirements have increased drastically. Thus, serving predictions from these models to meet any target latency and cost requirements of applications remains a key challenge, despite recent work in building inference-serving systems as well as algorithmic approaches that dynamically adapt models based on inputs. In this paper, we introduce a form of dynamism, modality selection, where we adaptively choose modalities from inference inputs while maintaining the model quality. We introduce MOSEL, an automated inference serving system for multi-modal ML models that carefully picks input modalities per request based on user-defined performance and accuracy requirements. MOSEL exploits modality configurations extensively, improving system throughput by 3.6$\times$ with an accuracy guarantee and shortening job completion times by 11$\times$.