CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models
This addresses the lack of systematic evaluation for composable AI methods, which is crucial for researchers and practitioners aiming to scale AI solutions for complex real-world problems, though it is incremental as it focuses on benchmarking rather than novel algorithmic advances.
The authors introduced CABENCH, the first public benchmark with 70 realistic composable AI tasks and 700 models, to systematically evaluate methods for solving complex tasks by composing ready-to-use models, establishing initial baselines with human-designed solutions and LLM-based approaches.
Composable AI offers a scalable and effective paradigm for tackling complex AI tasks by decomposing them into sub-tasks and solving each sub-task using ready-to-use well-trained models. However, systematically evaluating methods under this setting remains largely unexplored. In this paper, we introduce CABENCH, the first public benchmark comprising 70 realistic composable AI tasks, along with a curated pool of 700 models across multiple modalities and domains. We also propose an evaluation framework to enable end-to-end assessment of composable AI solutions. To establish initial baselines, we provide human-designed reference solutions and compare their performance with two LLM-based approaches. Our results illustrate the promise of composable AI in addressing complex real-world problems while highlighting the need for methods that can fully unlock its potential by automatically generating effective execution pipelines.