Resource-efficient Inference with Foundation Model Programs
This addresses resource efficiency for production deployments of multi-modal AI systems, offering a scalable solution with incremental improvements in cost reduction.
The paper tackles the high inference-time resource costs of large language and vision models by proposing foundation model programs that allocate smaller, cheaper models for simpler subtasks and larger models for complex ones, achieving up to 98% resource savings with minimal accuracy loss on streaming visual question-answering tasks.
The inference-time resource costs of large language and vision models present a growing challenge in production deployments. We propose the use of foundation model programs, i.e., programs that can invoke foundation models with varying resource costs and performance, as an approach to this problem. Specifically, we present a method that translates a task into a program, then learns a policy for resource allocation that, on each input, selects foundation model "backends" for each program module. The policy uses smaller, cheaper backends to handle simpler subtasks, while allowing more complex subtasks to leverage larger, more capable models. We evaluate the method on two new "streaming" visual question-answering tasks in which a system answers a question on a sequence of inputs, receiving ground-truth feedback after each answer. Compared to monolithic multi-modal models, our implementation achieves up to 98% resource savings with minimal accuracy loss, demonstrating its potential for scalable and resource-efficient multi-modal inference.