Federated Inference for Heterogeneous LLM Communication and Collaboration
This addresses the problem of limited on-device LLM capabilities for users needing efficient, privacy-preserving collaboration, but it is a position paper proposing a new paradigm rather than a full implementation.
The paper tackles the challenge of improving on-device Large Language Model (LLM) performance and efficiency by enabling collaboration through sharing data, tokens, and model weights, constrained by QoS, privacy, and heterogeneity, and presents a federated inference framework called FedRefine that allows heterogeneous LLMs to perform inference collaboratively while preserving privacy, with numerical results showing its superiority.
Given the limited performance and efficiency of on-device Large Language Models (LLMs), the collaborations between multiple LLMs enable desirable performance enhancements, in which data, tokens, and model weights could be shared across LLMs. This process is constrained by task-oriented QoS demands, privacy requirements, and inherent system heterogeneity. In view of the above challenge and to fully exploit the on-device inference capabilities, we present a novel federated inference framework in this position paper, termed federated refinement \texttt{FedRefine}. This framework presents a new paradigm for heterogeneous LLMs collaboratively performing inference with communicating KV caches in a privacy-preserving manner. Some numerical results are provided to highlight the superiority of \texttt{FedRefine}. Several interesting topics are also highlighted for future research. By exploring the LLM-native communications, we wish to provide a new paradigm for this broad area.