Aggregating Dependent Gaussian Experts in Local Approximation
This work addresses a key limitation in scaling Gaussian processes to large datasets for practitioners in machine learning, though it is incremental as it builds on existing DGP methods.
The paper tackles the problem of sub-optimal and inconsistent solutions in distributed Gaussian processes (DGPs) due to violations of the conditional independence assumption among local experts. It proposes a novel aggregation method using a Gaussian graphical model to detect dependencies, resulting in improved performance and time efficiency compared to state-of-the-art DGP approaches.
Distributed Gaussian processes (DGPs) are prominent local approximation methods to scale Gaussian processes (GPs) to large datasets. Instead of a global estimation, they train local experts by dividing the training set into subsets, thus reducing the time complexity. This strategy is based on the conditional independence assumption, which basically means that there is a perfect diversity between the local experts. In practice, however, this assumption is often violated, and the aggregation of experts leads to sub-optimal and inconsistent solutions. In this paper, we propose a novel approach for aggregating the Gaussian experts by detecting strong violations of conditional independence. The dependency between experts is determined by using a Gaussian graphical model, which yields the precision matrix. The precision matrix encodes conditional dependencies between experts and is used to detect strongly dependent experts and construct an improved aggregation. Using both synthetic and real datasets, our experimental evaluations illustrate that our new method outperforms other state-of-the-art (SOTA) DGP approaches while being substantially more time-efficient than SOTA approaches, which build on independent experts.