Information Bottleneck Methods for Distributed Learning
This work addresses efficient data compression in distributed learning systems, but it is incremental as it builds on existing information bottleneck theory with specific extensions to streaming data.
The authors tackled the problem of compressing training data for distributed learning by framing it as a rate-distortion problem, showing that information bottleneck methods work for batch data and introducing a new algorithm for streaming Gaussian data with a two-pass improvement. They derived the rate dependency on sample size for optimal cross-entropy loss scaling.
We study a distributed learning problem in which Alice sends a compressed distillation of a set of training data to Bob, who uses the distilled version to best solve an associated learning problem. We formalize this as a rate-distortion problem in which the training set is the source and Bob's cross-entropy loss is the distortion measure. We consider this problem for unsupervised learning for batch and sequential data. In the batch data, this problem is equivalent to the information bottleneck (IB), and we show that reduced-complexity versions of standard IB methods solve the associated rate-distortion problem. For the streaming data, we present a new algorithm, which may be of independent interest, that solves the rate-distortion problem for Gaussian sources. Furthermore, to improve the results of the iterative algorithm for sequential data we introduce a two-pass version of this algorithm. Finally, we show the dependency of the rate on the number of samples $k$ required for Gaussian sources to ensure cross-entropy loss that scales optimally with the growth of the training set.