Cornserve: Efficiently Serving Any-to-Any Multimodal Models
This addresses the problem of high computational overhead in serving complex multimodal models for AI developers and users, though it is incremental as it builds on existing serving systems.
The paper tackles the challenge of efficiently serving Any-to-Any multimodal models, which handle heterogeneous input/output combinations, by introducing Cornserve, a system that automatically plans and executes optimized deployments, achieving up to 3.81x throughput improvement and 5.79x tail latency reduction over existing solutions.
We present Cornserve, an efficient online serving system for an emerging class of multimodal models called Any-to-Any models. Any-to-Any models accept combinations of text and multimodal data (e.g., image, video, audio) as input and also generate combinations of text and multimodal data as output, introducing request type, computation path, and computation scaling heterogeneity in model serving. Cornserve allows model developers to describe the computation graph of generic Any-to-Any models, which consists of heterogeneous components such as multimodal encoders, autoregressive models like Large Language Models (LLMs), and multimodal generators like Diffusion Transformers (DiTs). Given this, Cornserve's planner automatically finds an optimized deployment plan for the model, including whether and how to disaggregate the model into smaller components based on model and workload characteristics. Cornserve's distributed runtime then executes the model per the plan, efficiently handling Any-to-Any model heterogeneity during online serving. Evaluations show that Cornserve can efficiently serve diverse Any-to-Any models and workloads, delivering up to 3.81$\times$ throughput improvement and up to 5.79$\times$ tail latency reduction over existing solutions.