Accelerating Multi-Model Inference by Merging DNNs of Different Weights
This addresses efficiency issues for AI practitioners deploying multiple fine-tuned models, though it is incremental as it builds on existing model merging concepts.
The paper tackles the problem of low GPU utilization when serving multiple fine-tuned DNN models with different weights, which prevents batching, by proposing NetFuse to merge such models, achieving speedups of up to 3.6x on a V100 GPU with minimal memory overhead.
Standardized DNN models that have been proved to perform well on machine learning tasks are widely used and often adopted as-is to solve downstream tasks, forming the transfer learning paradigm. However, when serving multiple instances of such DNN models from a cluster of GPU servers, existing techniques to improve GPU utilization such as batching are inapplicable because models often do not share weights due to fine-tuning. We propose NetFuse, a technique of merging multiple DNN models that share the same architecture but have different weights and different inputs. NetFuse is made possible by replacing operations with more general counterparts that allow a set of weights to be associated with only a certain set of inputs. Experiments on ResNet-50, ResNeXt-50, BERT, and XLNet show that NetFuse can speed up DNN inference time up to 3.6x on a NVIDIA V100 GPU, and up to 3.0x on a TITAN Xp GPU when merging 32 model instances, while only using up a small additional amount of GPU memory.