How to Purchase Labels? A Cost-Effective Approach Using Active Learning Markets
This provides a practical solution for analysts in resource-constrained environments, such as real estate pricing and energy forecasting, to optimize data acquisition, though it is incremental as it builds on existing active learning strategies.
The paper tackles the problem of cost-effective label acquisition for model improvement by introducing active learning markets, which consistently achieve superior performance with fewer labels compared to conventional methods like random sampling and greedy heuristics.
We introduce and analyse active learning markets as a way to purchase labels, in situations where analysts aim to acquire additional data to improve model fitting, or to better train models for predictive analytics applications. This comes in contrast to the many proposals that already exist to purchase features and examples. By originally formalising the market clearing as an optimisation problem, we integrate budget constraints and improvement thresholds into the label acquisition process. We focus on a single-buyer-multiple-seller setup and propose the use of two active learning strategies (variance based and query-by-committee based), paired with distinct pricing mechanisms. They are compared to benchmark baselines including random sampling and a greedy knapsack heuristic. The proposed strategies are validated on real-world datasets from two critical application domains: real estate pricing and energy forecasting. Results demonstrate the robustness of our approach, consistently achieving superior performance with fewer labels acquired compared to conventional methods. Our proposal comprises an easy-to-implement practical solution for optimising data acquisition in resource-constrained environments.