Benchmarking Robust Aggregation in Decentralized Gradient Marketplaces
This work addresses the need for better evaluation tools in decentralized machine learning marketplaces, offering incremental improvements by extending existing FL benchmarks to include marketplace-specific dimensions.
The paper tackles the problem of evaluating robust gradient aggregation methods in decentralized gradient marketplaces, which lack benchmarks for economic and systemic factors like cost-effectiveness and fairness, by introducing a comprehensive benchmark framework and providing empirical analysis of existing methods like MartFL with adapted strategies such as FLTrust and SkyMask across diverse datasets and attacks.
The rise of distributed and privacy-preserving machine learning has sparked interest in decentralized gradient marketplaces, where participants trade intermediate artifacts like gradients. However, existing Federated Learning (FL) benchmarks overlook critical economic and systemic factors unique to such marketplaces-cost-effectiveness, fairness to sellers, and market stability-especially when a buyer relies on a private baseline dataset for evaluation. We introduce a comprehensive benchmark framework to holistically evaluate robust gradient aggregation methods within these buyer-baseline-reliant marketplaces. Our contributions include: (1) a simulation environment modeling marketplace dynamics with a variable buyer baseline and diverse seller distributions; (2) an evaluation methodology augmenting standard FL metrics with marketplace-centric dimensions such as Economic Efficiency, Fairness, and Selection Dynamics; (3) an in-depth empirical analysis of the existing Distributed Gradient Marketplace framework, MartFL, including the integration and comparative evaluation of adapted FLTrust and SkyMask as alternative aggregation strategies within it. This benchmark spans diverse datasets, local attacks, and Sybil attacks targeting the marketplace selection process; and (4) actionable insights into the trade-offs between model performance, robustness, cost, fairness, and stability. This benchmark equips the community with essential tools and empirical evidence to evaluate and design more robust, equitable, and economically viable decentralized gradient marketplaces.