Federated Representation Learning via Maximal Coding Rate Reduction
This addresses the challenge of efficient and effective representation learning in decentralized settings for applications like privacy-preserving machine learning, though it is incremental as it adapts an existing principle to federated learning.
The paper tackles the problem of learning low-dimensional representations from distributed data in federated learning by replacing the cross-entropy loss with maximal coding rate reduction (MCR2), resulting in representations that are discriminative and compressible, with theoretical guarantees of convergence to a stationary point and demonstrated utility in experiments.
We propose a federated methodology to learn low-dimensional representations from a dataset that is distributed among several clients. In particular, we move away from the commonly-used cross-entropy loss in federated learning, and seek to learn shared low-dimensional representations of the data in a decentralized manner via the principle of maximal coding rate reduction (MCR2). Our proposed method, which we refer to as FLOW, utilizes MCR2 as the objective of choice, hence resulting in representations that are both between-class discriminative and within-class compressible. We theoretically show that our distributed algorithm achieves a first-order stationary point. Moreover, we demonstrate, via numerical experiments, the utility of the learned low-dimensional representations.