Adaptive Contracts for Cost-Effective AI Delegation
This work addresses cost-effectiveness for organizations delegating AI tasks, offering an incremental improvement in contract design to balance evaluation accuracy and expenses.
The paper tackles the problem of rising costs in pay-for-performance AI delegation contracts due to noisy evaluation, by introducing adaptive contracts that selectively use detailed evaluation after an initial coarse signal to conserve resources. It provides algorithms for optimal contracts, proves hardness results, and empirically shows benefits over non-adaptive baselines on question-answering and code-generation datasets.
When organizations delegate text generation tasks to AI providers via pay-for-performance contracts, expected payments rise when evaluation is noisy. As evaluation methods become more elaborate, the economic benefits of decreased noise are often overshadowed by increased evaluation costs. In this work, we introduce adaptive contracts for AI delegation, which allow detailed evaluation to be performed selectively after observing an initial coarse signal in order to conserve resources. We make three sets of contributions: First, we provide efficient algorithms for computing optimal adaptive contracts under natural assumptions or when core problem dimensions are small, and prove hardness of approximation in the general unstructured case. We then formulate alternative models of randomized adaptive contracts and discuss their benefits and limitations. Finally, we empirically demonstrate the benefits of adaptivity over non-adaptive baselines using question-answering and code-generation datasets.