Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing
This addresses the problem of cost-efficient and transparent LLM routing for practitioners, representing an incremental improvement over existing black-box systems.
The paper tackles the problem of selecting the right LLM for a task without wasting money by introducing BELLA, a framework that recommends optimal LLM selection through interpretable skill-based profiling, enabling cost-performance trade-offs for practitioners.
How should Large Language Model (LLM) practitioners select the right model for a task without wasting money? We introduce BELLA (Budget-Efficient LLM Selection via Automated skill-profiling), a framework that recommends optimal LLM selection for tasks through interpretable skill-based model selection. Standard benchmarks report aggregate metrics that obscure which specific capabilities a task requires and whether a cheaper model could suffice. BELLA addresses this gap through three stages: (1) decomposing LLM outputs and extract the granular skills required by using critic-based profiling, (2) clustering skills into structured capability matrices, and (3) multi-objective optimization to select the right models to maximize performance while respecting budget constraints. BELLA provides natural-language rationale for recommendations, providing transparency that current black-box routing systems lack. We describe the framework architecture, situate it within the landscape of LLM routing and evaluation, and discuss its application to financial reasoning as a representative domain exhibiting diverse skill requirements and cost-variation across models. Our framework enables practitioners to make principled and cost-performance trade-offs for deploying LLMs.