Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study
This work addresses the need for standardized benchmarking in FMTL, which is crucial for researchers and practitioners applying collaborative learning to multi-task datasets with non-IID data, though it is incremental as it focuses on evaluation rather than new algorithms.
The paper tackles the lack of a comprehensive evaluation method for Federated Multi-Task Learning (FMTL) by introducing FMTL-Bench, a benchmark framework for systematic evaluation across data, model, and optimization levels, including experiments on non-IID data scenarios and case studies on communication, time, and energy consumption.
The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code can be found on https://github.com/youngfish42/FMTL-Benchmark .