FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
This work addresses the problem of inconsistent evaluation for researchers and practitioners in deep learning, providing a standardized tool for benchmarking fusion methods, though it is incremental as it builds on existing techniques without introducing new fusion methods.
The authors tackled the inconsistent and inadequate evaluation of deep model fusion techniques by introducing FusionBench, the first unified library and comprehensive benchmark for deep model fusion, which includes multiple tasks with various models and datasets to enable consistent comparisons across scenarios and model scales.
Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single better-performing model in a cost-effective and data-efficient manner. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness. We present FusionBench, the first benchmark and a unified library designed specifically for deep model fusion. Our benchmark consists of multiple tasks, each with different settings of models and datasets. This variety allows us to compare fusion methods across different scenarios and model scales. Additionally, FusionBench serves as a unified library for easy implementation and testing of new fusion techniques. FusionBench is open source and actively maintained, with community contributions encouraged. Homepage https://github.com/tanganke/fusion_bench