Inductive biases and Self Supervised Learning in modelling a physical heating system
This work addresses the need for efficient and accurate models in control systems with delayed responses, though it is incremental as it builds on existing self-supervised learning and neural network techniques.
The authors tackled the problem of modeling a physical heating system with noise and inertia for Model Predictive Control by inferring inductive biases and designing a new neural network architecture called Delay, which achieved comparable prediction performance to baseline attention-based recurrent networks while being faster and better at exploiting larger data volumes.
Model Predictive Controllers (MPC) require a good model for the controlled process. In this paper I infer inductive biases about a physical system. I use these biases to derive a new neural network architecture that can model this real system that has noise and inertia. The main inductive biases exploited here are: the delayed impact of some inputs on the system and the separability between the temporal component and how the inputs interact to produce the output of a system. The inputs are independently delayed using shifted convolutional kernels. Feature interactions are modelled using a fully connected network that does not have access to temporal information. The available data and the problem setup allow the usage of Self Supervised Learning in order to train the models. The baseline architecture is an Attention based Reccurent network adapted to work with MPC like inputs. The proposed networks are faster, better at exploiting larger data volumes and are almost as good as baseline networks in terms of prediction performance. The proposed architecture family called Delay can be used in a real scenario to control systems with delayed responses with respect to its controls or inputs. Ablation studies show that the presence of delay kernels are vital to obtain any learning in proposed architecture. Code and some experimental data are available online.