QueryMarket: Cost-Aware Online Active Learning in Data Markets
For analysts working with real-time data streams, this work addresses the practical problem of purchasing labels under budget constraints and concept drift, though the improvements are incremental over existing online active learning methods.
QueryMarket introduces a cost-aware online active learning framework that integrates data pricing, information gain, and rolling budget constraints under concept drift. The proposed OVBAL method achieves a more favorable long-run error-cost trade-off in real-world solar power forecasting, particularly under seller-centric pricing.
Data acquisition is a major bottleneck for learning in real-time streams: analysts must decide on the fly which labels to purchase while respecting a rolling budget. However, existing online active learning rarely unifies pricing, information gain, and rolling budget constraints under concept drift. We introduce QueryMarket, a market-inspired framework that queries each incoming data point based on its estimated utility to the model and its price. Within this framework, we propose OVBAL (online variance-based active learning), which integrates data pricing with information-driven selection by estimating each sample's marginal utility via a D-optimality criterion with exponential forgetting and executing cost-aware purchases under rolling budget constraints. OVBAL yields a simple, fully online decision rule that adapts to nonstationary streams and heterogeneous label costs. Experiments on synthetic data and a real-world solar power generation forecasting task show that OVBAL is particularly effective under seller-centric pricing and yields a more favorable long-run error-cost trade-off in the real-world task under both pricing schemes.