Learning of Gaussian Processes in Distributed and Communication Limited Systems
This work addresses communication efficiency for distributed statistical learning, but it is incremental as it builds on existing GP methods with new quantization strategies.
The paper tackles the problem of learning Gaussian Processes in distributed systems with limited communication, showing that their proposed methods, using few bits per symbol, outperform previous zero-rate distributed GP learning schemes like Bayesian Committee Model and Product of Experts.
It is of fundamental importance to find algorithms obtaining optimal performance for learning of statistical models in distributed and communication limited systems. Aiming at characterizing the optimal strategies, we consider learning of Gaussian Processes (GPs) in distributed systems as a pivotal example. We first address a very basic problem: how many bits are required to estimate the inner-products of Gaussian vectors across distributed machines? Using information theoretic bounds, we obtain an optimal solution for the problem which is based on vector quantization. Two suboptimal and more practical schemes are also presented as substitute for the vector quantization scheme. In particular, it is shown that the performance of one of the practical schemes which is called per-symbol quantization is very close to the optimal one. Schemes provided for the inner-product calculations are incorporated into our proposed distributed learning methods for GPs. Experimental results show that with spending few bits per symbol in our communication scheme, our proposed methods outperform previous zero rate distributed GP learning schemes such as Bayesian Committee Model (BCM) and Product of experts (PoE).