Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates
This work addresses communication efficiency for federated learning systems, but it is incremental as it builds upon existing federated distillation methods.
The paper tackled the problem of high communication overhead in federated distillation by proposing ALU, a technique that accumulates local updates before transferring knowledge, which drastically reduces communication frequency and overhead.
As an emerging federated learning paradigm, federated distillation enables communication-efficient model training by transmitting only small-scale knowledge during the learning process. To further improve the communication efficiency of federated distillation, we propose a novel technique, ALU, which accumulates multiple rounds of local updates before transferring the knowledge to the central server. ALU drastically decreases the frequency of communication in federated distillation, thereby significantly reducing the communication overhead during the training process. Empirical experiments demonstrate the substantial effect of ALU in improving the communication efficiency of federated distillation.