Learning to Assign Prediction Tasks to Agents with Capacity Constraints
For system designers managing multiple prediction agents (humans or AI) with limited capacity, this work provides a principled framework for learning optimal task assignment policies.
This paper addresses the problem of assigning prediction tasks to agents with capacity constraints, proposing sequential explore-exploit algorithms that learn agent expertise and assignment policies. Experiments across tabular, image, and text tasks show systematic gains over non-contextual baselines for both LLM and human agents.
We address the problem of learning to assign prediction tasks to one agent from a set of available human or AI agents. In particular, we focus on the sequential learning of agent expertise and assignment policies where each agent is constrained to handle a fraction of tasks. We provide a general theoretical characterization of this problem in terms of agent capacities, differences in agent expertise, and task context. We then develop a framework of sequential explore-exploit policy-learning algorithms that seek to maximize overall performance. Experimental results over a variety of tabular, image, and text prediction tasks demonstrate systematic gains from our policy-learning algorithms relative to non-contextual baselines across different types of agents, including LLMs and humans.