Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel
This work addresses the problem of guiding neural network learning with known properties like seasonality, particularly for applications such as remote sensing, though it appears incremental as it combines existing methods.
The authors tackled the challenge of incorporating prior knowledge into neural networks by proposing an Implicit Composite Kernel (ICK) that blends neural networks with Gaussian processes, demonstrating superior performance and flexibility on synthetic and real-world datasets.
It is challenging to guide neural network (NN) learning with prior knowledge. In contrast, many known properties, such as spatial smoothness or seasonality, are straightforward to model by choosing an appropriate kernel in a Gaussian process (GP). Many deep learning applications could be enhanced by modeling such known properties. For example, convolutional neural networks (CNNs) are frequently used in remote sensing, which is subject to strong seasonal effects. We propose to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties (e.g., seasonality). We implement this idea by combining a deep network and an efficient mapping based on the Nystrom approximation, which we call Implicit Composite Kernel (ICK). We then adopt a sample-then-optimize approach to approximate the full GP posterior distribution. We demonstrate that ICK has superior performance and flexibility on both synthetic and real-world data sets. We believe that ICK framework can be used to include prior information into neural networks in many applications.