Benchmarking Compound AI Applications for Hardware-Software Co-Design
This work addresses a gap for the systems community by providing a tool to guide hardware-software co-design in datacenters, though it is incremental as it builds on existing concepts of benchmarking and co-design.
The paper tackles the lack of standardized benchmarks for analyzing the complex design-space of Compound AI applications, which involve interactions between LLMs, ML models, tools, and data sources, by presenting a benchmarking suite that enables cross-stack analysis and derives design principles for hardware-software co-design to improve resource-efficiency.
Compound AI applications, composed from interactions between Large Language Models (LLMs), Machine Learning (ML) models, external tools and data sources are quickly becoming an integral workload in datacenters. Their diverse sub-components and use-cases present a large configuration-space across the deployment stack -- ranging from applications and serving software down to hardware -- each of which may influence the application performance, deployment cost, and/or resource consumption. Despite their rapid adoption, however, the systems community lacks a standardized benchmark for analyzing this complicated design-space and guiding in system design. In this work, we present our benchmarking suite used for cross-stack analysis of Compound AI applications. Using this, we derive key takeaways and design principles spanning several layers of the stack for hardware-software co-design to unlock higher resource-efficiency.