Federated Neural Compression Under Heterogeneous Data
This work addresses efficient data compression for distributed systems with non-i.i.d. data, but it is incremental as it builds on existing personalized federated learning techniques.
The paper tackles the problem of learning a compressor from heterogeneous data distributed across clients in a federated setting, and shows that using a shared global representation with personalized entropy models outperforms purely local methods.
We discuss a federated learned compression problem, where the goal is to learn a compressor from real-world data which is scattered across clients and may be statistically heterogeneous, yet share a common underlying representation. We propose a distributed source model that encompasses both characteristics, and naturally suggests a compressor architecture that uses analysis and synthesis transforms shared by clients. Inspired by personalized federated learning methods, we employ an entropy model that is personalized to each client. This allows for a global latent space to be learned across clients, and personalized entropy models that adapt to the clients' latent distributions. We show empirically that this strategy outperforms solely local methods, which indicates that learned compression also benefits from a shared global representation in statistically heterogeneous federated settings.