VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks
This work addresses the problem of inadequate benchmarking for VFL researchers, though it is incremental as it builds on existing VFL methods by enhancing evaluation tools.
The paper tackled the lack of diverse feature distributions in Vertical Federated Learning (VFL) benchmarks by introducing factors like feature importance and correlation, along with new metrics and a real dataset for image-image VFL, leading to improved algorithm evaluation.
Vertical Federated Learning (VFL) is a crucial paradigm for training machine learning models on feature-partitioned, distributed data. However, due to privacy restrictions, few public real-world VFL datasets exist for algorithm evaluation, and these represent a limited array of feature distributions. Existing benchmarks often resort to synthetic datasets, derived from arbitrary feature splits from a global set, which only capture a subset of feature distributions, leading to inadequate algorithm performance assessment. This paper addresses these shortcomings by introducing two key factors affecting VFL performance - feature importance and feature correlation - and proposing associated evaluation metrics and dataset splitting methods. Additionally, we introduce a real VFL dataset to address the deficit in image-image VFL scenarios. Our comprehensive evaluation of cutting-edge VFL algorithms provides valuable insights for future research in the field.