A Large-Scale Web Search Dataset for Federated Online Learning to Rank
This addresses privacy concerns in search ranking for researchers by providing a more realistic dataset, though it is incremental as it builds on existing FOLTR methods.
The paper tackles the lack of realistic benchmarks in Federated Online Learning to Rank (FOLTR) by introducing AOL4FOLTR, a large-scale dataset with 2.6 million queries from 10,000 users, which includes real click data and timestamps to enable more accurate simulations.
The centralized collection of search interaction logs for training ranking models raises significant privacy concerns. Federated Online Learning to Rank (FOLTR) offers a privacy-preserving alternative by enabling collaborative model training without sharing raw user data. However, benchmarks in FOLTR are largely based on random partitioning of classical learning-to-rank datasets, simulated user clicks, and the assumption of synchronous client participation. This oversimplifies real-world dynamics and undermines the realism of experimental results. We present AOL4FOLTR, a large-scale web search dataset with 2.6 million queries from 10,000 users. Our dataset addresses key limitations of existing benchmarks by including user identifiers, real click data, and query timestamps, enabling realistic user partitioning, behavior modeling, and asynchronous federated learning scenarios.