Giulio Bottegal

Riccardo Sven Risuleo, Marco Molinari, Giulio Bottegal et al.

This paper describes a benchmark consisting of a set of synthetic measurements relative to an office environment simulated with the software IDA-ICE. The simulated environment reproduces a laboratory at the KTH-EES Smart Building, equipped with a building management system. The data set contains records collected over a period of several days. The signals to CO$_2$ concentration, mechanical ventilation airflows, air infiltrations and occupancy. Information on door and window opening is also available. This benchmark is intended for testing data-based modeling techniques. The ultimate goal is the development of models to improve the forecast and control of environmental variables. Among the numerous challenges related to this framework, we point out the problem of occupancy estimation using information on CO$_2$ concentration. This can be seen as a blind identification problem. For benchmarking purposes, we present two different identification approaches: a baseline overparametrization method and a kernel-based method.

Niklas Everitt, Giulio Bottegal, Håkan Hjalmarsson

We present a new method of identifying a specific module in a dynamic network, possibly with feedback loops. Assuming known topology, we express the dynamics by an acyclic network composed of two blocks where the first block accounts for the relation between the known reference signals and the input to the target module, while the second block contains the target module. Using an empirical Bayes approach, we model the first block as a Gaussian vector with covariance matrix (kernel) given by the recently introduced stable spline kernel. The parameters of the target module are estimated by solving a marginal likelihood problem with a novel iterative scheme based on the Expectation-Maximization algorithm. Additionally, we extend the method to include additional measurements downstream of the target module. Using Markov Chain Monte Carlo techniques, it is shown that the same iterative scheme can solve also this formulation. Numerical experiments illustrate the effectiveness of the proposed methods.

Riccardo Sven Risuleo, Giulio Bottegal, Håkan Hjalmarsson

In this paper, we propose a novel algorithm for the identification of Hammerstein systems. Adopting a Bayesian approach, we model the impulse response of the unknown linear dynamic system as a realization of a zero-mean Gaussian process. The covariance matrix (or kernel) of this process is given by the recently introduced stable-spline kernel, which encodes information on the stability and regularity of the impulse response. The static non-linearity of the model is identified using an Empirical Bayes approach, i.e. by maximizing the output marginal likelihood, which is obtained by integrating out the unknown impulse response. The related optimization problem is solved adopting a novel iterative scheme based on the Expectation-Maximization (EM) method, where each iteration consists in a simple sequence of update rules. Numerical experiments show that the proposed method compares favorably with a standard algorithm for Hammerstein system identification.

Shengling Shi, Giulio Bottegal, Paul M. J. Van den Hof

In networks of dynamic systems, one challenge is to identify the interconnection structure on the basis of measured signals. Inspired by a Bayesian approach in [1], in this paper, we explore a Bayesian model selection method for identifying the connectivity of networks of transfer functions, without the need to estimate the dynamics. The algorithm employs a Bayesian measure and a forward-backward search algorithm. To obtain the Bayesian measure, the impulse responses of network modules are modeled as Gaussian processes and the hyperparameters are estimated by marginal likelihood maximization using the expectation-maximization algorithm. Numerical results demonstrate the effectiveness of this method.

Paul M. J. Van den Hof, Karthik R. Ramaswamy, Arne G. Dankers et al.

The identification of local modules in dynamic networks with known topology has recently been addressed by formulating conditions for arriving at consistent estimates of the module dynamics, typically under the assumption of having disturbances that are uncorrelated over the different nodes. The conditions typically reflect the selection of a set of node signals that are taken as predictor inputs in a MISO identification setup. In this paper an extension is made to arrive at an identification setup for the situation that process noises on the different node signals can be correlated with each other. In this situation the local module may need to be embedded in a MIMO identification setup for arriving at a consistent estimate with maximum likelihood properties. This requires the proper treatment of confounding variables. The result is an algorithm that, based on the given network topology and disturbance correlation structure, selects an appropriate set of node signals as predictor inputs and outputs in a MISO or MIMO identification setup. As a first step in the analysis, we restrict attention to the (slightly conservative) situation where the selected output node signals are predicted based on all of their in-neighbor node signals in the network.

Riccardo Sven Risuleo, Giulio Bottegal, Håkan Hjalmarsson

In this contribution, we propose a kernel-based method for the identification of linear systems from noisy and incomplete input-output datasets. We model the impulse response of the system as a Gaussian process whose covariance matrix is given by the recently introduced stable spline kernel. We adopt an empirical Bayes approach to estimate the posterior distribution of the impulse response given the data. The noiseless and missing data samples, together with the kernel hyperparameters, are estimated maximizing the joint marginal likelihood of the input and output measurements. To compute the marginal-likelihood maximizer, we build a solution scheme based on the Expectation-Maximization method. Simulations on a benchmark dataset show the effectiveness of the method.

Giulio Bottegal, Giorgio Picci

We propose a new modeling paradigm for large dimensional aggregates of stochastic systems by Generalized Factor Analysis (GFA) models. These models describe the data as the sum of a flocking plus an uncorrelated idiosyncratic component. The flocking component describes a sort of collective orderly motion which admits a much simpler mathematical description than the whole ensemble while the idiosyncratic component describes weakly correlated noise. We first discuss static GFA representations and characterize in a rigorous way the properties of the two components. The extraction of the dynamic flocking component is discussed for time-stationary linear systems and for a simple classes of separable random fields.

Karthik R. Ramaswamy, Giulio Bottegal, Paul M. J. Van den Hof

In order to identify a system (module) embedded in a dynamic network, one has to formulate a multiple-input estimation problem that necessitates certain nodes to be measured and included as predictor inputs. However, some of these nodes may not be measurable in many practical cases due to sensor selection and placement issues. This may result in biased estimates of the target module. Furthermore, the identification problem associated with the multiple-input structure may require determining a large number of parameters that are not of particular interest to the experimenter, with increased computational complexity in large-sized networks. In this paper, we tackle these problems by using a data augmentation strategy that allows us to reconstruct the missing node measurements and increase the accuracy of the estimated target module. To this end, we develop a system identification method using regularized kernel-based methods coupled with approximate inference methods. Keeping a parametric model for the module of interest, we model the other modules as Gaussian Processes (GP) with a kernel given by the so-called stable spline kernel. An Empirical Bayes (EB) approach is used to estimate the parameters of the target module. The related optimization problem is solved using an Expectation-Maximization (EM) method, where we employ a Markov-chain Monte Carlo (MCMC) technique to reconstruct the unknown missing node information and the network dynamics. Numerical simulations on dynamic network examples illustrate the potentials of the developed method.

Riccardo Sven Risuleo, Giulio Bottegal, Håkan Hjalmarsson

In this work, we present a new class of models, called uncertain-input models, that allows us to treat system-identification problems in which a linear system is subject to a partially unknown input signal. To encode prior information about the input or the linear system, we use Gaussian-process models. We estimate the model from data using the empirical Bayes approach: the input and the impulse responses of the linear system are estimated using the posterior means of the Gaussian-process models given the data, and the hyperparameters that characterize the Gaussian-process models are estimated from the marginal likelihood of the data. We propose an iterative algorithm to find the hyperparameters that relies on the EM method and results in simple update steps. In the most general formulation, neither the marginal likelihood nor the posterior distribution of the unknowns is tractable. Therefore, we propose two approximation approaches, one based on Markov-chain Monte Carlo techniques and one based on variational Bayes approximation. We also show special model structures for which the distributions are treatable exactly. Through numerical simulations, we study the application of the uncertain-input model to the identification of Hammerstein systems and cascaded linear systems. As part of the contribution of the paper, we show that this model structure encompasses many classical problems in system identification such as classical PEM, Hammerstein models, errors-in-variables problems, blind system identification, and cascaded linear systems. This allows us to build a systematic procedure to apply the algorithms proposed in this work to a wide class of classical problems.

Giulio Bottegal, Gianluigi Pillonetto

Generalized cross validation (GCV) is one of the most important approaches used to estimate parameters in the context of inverse problems and regularization techniques. A notable example is the determination of the smoothness parameter in splines. When the data are generated by a state space model, like in the spline case, efficient algorithms are available to evaluate the GCV score with complexity that scales linearly in the data set size. However, these methods are not amenable to on-line applications since they rely on forward and backward recursions. Hence, if the objective has been evaluated at time $t-1$ and new data arrive at time t, then O(t) operations are needed to update the GCV score. In this paper we instead show that the update cost is $O(1)$, thus paving the way to the on-line use of GCV. This result is obtained by deriving the novel GCV filter which extends the classical Kalman filter equations to efficiently propagate the GCV score over time. We also illustrate applications of the new filter in the context of state estimation and on-line regularized linear system identification.

Giulio Bottegal, Farhad Farokhi, Iman Shames

Adding input and output noises for increasing model identification error of finite impulse response (FIR) systems is considered. This is motivated by the desire to protect the model of the system as a trade secret by rendering model identification techniques ineffective. Optimal filters for constructing additive noises that maximizes the identification error subject to maintaining the closed-loop performance degradation below a limit are constructed. Furthermore, differential privacy is used for designing output noises that preserve the privacy of the model.

Giulio Bottegal, Håkan Hjalmarsson, Gianluigi Pillonetto

In this paper we introduce a novel method for linear system identification with quantized output data. We model the impulse response as a zero-mean Gaussian process whose covariance (kernel) is given by the recently proposed stable spline kernel, which encodes information on regularity and exponential stability. This serves as a starting point to cast our system identification problem into a Bayesian framework. We employ Markov Chain Monte Carlo methods to provide an estimate of the system. In particular, we design two methods based on the so-called Gibbs sampler that allow also to estimate the kernel hyperparameters by marginal likelihood maximization via the expectation-maximization method. Numerical simulations show the effectiveness of the proposed scheme, as compared to the state-of-the-art kernel-based methods when these are employed in system identification with quantized data.

Aleksandr Y. Aravkin, Giulio Bottegal, Gianluigi Pillonetto

Boosting combines weak (biased) learners to obtain effective learning algorithms for classification and prediction. In this paper, we show a connection between boosting and kernel-based methods, highlighting both theoretical and practical applications. In the context of $\ell_2$ boosting, we start with a weak linear learner defined by a kernel $K$. We show that boosting with this learner is equivalent to estimation with a special {\it boosting kernel} that depends on $K$, as well as on the regression matrix, noise variance, and hyperparameters. The number of boosting iterations is modeled as a continuous hyperparameter, and fit along with other parameters using standard techniques. We then generalize the boosting kernel to a broad new class of boosting approaches for more general weak learners, including those based on the $\ell_1$, hinge and Vapnik losses. The approach allows fast hyperparameter tuning for this general class, and has a wide range of applications, including robust regression and classification. We illustrate some of these applications with numerical examples on synthetic and real data.

Riccardo Sven Risuleo, Giulio Bottegal, Håkan Hjalmarsson

Recent developments in system identification have brought attention to regularized kernel-based methods, where, adopting the recently introduced stable spline kernel, prior information on the unknown process is enforced. This reduces the variance of the estimates and thus makes kernel-based methods particularly attractive when few input-output data samples are available. In such cases however, the influence of the system initial conditions may have a significant impact on the output dynamics. In this paper, we specifically address this point. We propose three methods that deal with the estimation of initial conditions using different types of information. The methods consist in various mixed maximum likelihood--a posteriori estimators which estimate the initial conditions and tune the hyperparameters characterizing the stable spline kernel. To solve the related optimization problems, we resort to the expectation-maximization method, showing that the solutions can be attained by iterating among simple update steps. Numerical experiments show the advantages, in terms of accuracy in reconstructing the system impulse response, of the proposed strategies, compared to other kernel-based schemes not accounting for the effect initial conditions.

Riccardo Sven Risuleo, Giulio Bottegal, Håkan Hjalmarsson

In this paper we propose a new identification scheme for Hammerstein systems, which are dynamic systems consisting of a static nonlinearity and a linear time-invariant dynamic system in cascade. We assume that the nonlinear function can be described as a linear combination of $p$ basis functions. We reconstruct the $p$ coefficients of the nonlinearity together with the first $n$ samples of the impulse response of the linear system by estimating an $np$-dimensional overparameterized vector, which contains all the combinations of the unknown variables. To avoid high variance in these estimates, we adopt a regularized kernel-based approach and, in particular, we introduce a new kernel tailored for Hammerstein system identification. We show that the resulting scheme provides an estimate of the overparameterized vector that can be uniquely decomposed as the combination of an impulse response and $p$ coefficients of the static nonlinearity. We also show, through several numerical experiments, that the proposed method compares very favorably with two standard methods for Hammerstein system identification.

Giulio Bottegal, Gianluigi Pillonetto, Håkan Hjalmarsson

In this paper we introduce a novel method for linear system identification with quantized output data. We model the impulse response as a zero-mean Gaussian process whose covariance (kernel) is given by the recently proposed stable spline kernel, which encodes information on regularity and exponential stability. This serves as a starting point to cast our system identification problem into a Bayesian framework. We employ Markov Chain Monte Carlo (MCMC) methods to provide an estimate of the system. In particular, we show how to design a Gibbs sampler which quickly converges to the target distribution. Numerical simulations show a substantial improvement in the accuracy of the estimates over state-of-the-art kernel-based methods when employed in identification of systems with quantized data.

Giulio Bottegal, Riccardo S. Risuleo, Håkan Hjalmarsson

We propose a new method for blind system identification. Resorting to a Gaussian regression framework, we model the impulse response of the unknown linear system as a realization of a Gaussian process. The structure of the covariance matrix (or kernel) of such a process is given by the stable spline kernel, which has been recently introduced for system identification purposes and depends on an unknown hyperparameter. We assume that the input can be linearly described by few parameters. We estimate these parameters, together with the kernel hyperparameter and the noise variance, using an empirical Bayes approach. The related optimization problem is efficiently solved with a novel iterative scheme based on the Expectation-Maximization method. In particular, we show that each iteration consists of a set of simple update rules. We show, through some numerical experiments, very promising performance of the proposed method.

Giulio Bottegal, Aleksandr Y. Aravkin, Håkan Hjalmarsson et al.

Recent developments in system identification have brought attention to regularized kernel-based methods. This type of approach has been proven to compare favorably with classic parametric methods. However, current formulations are not robust with respect to outliers. In this paper, we introduce a novel method to robustify kernel-based system identification methods. To this end, we model the output measurement noise using random variables with heavy-tailed probability density functions (pdfs), focusing on the Laplacian and the Student's t distributions. Exploiting the representation of these pdfs as scale mixtures of Gaussians, we cast our system identification problem into a Gaussian process regression framework, which requires estimating a number of hyperparameters of the data size order. To overcome this difficulty, we design a new maximum a posteriori (MAP) estimator of the hyperparameters, and solve the related optimization problem with a novel iterative scheme based on the Expectation-Maximization (EM) method. In presence of outliers, tests on simulated data and on a real system show a substantial performance improvement compared to currently used kernel-based methods for linear system identification.

Giulio Bottegal, Aleksandr Y. Aravkin, Hakan Hjalmarsson et al.

In this paper, we propose an outlier-robust regularized kernel-based method for linear system identification. The unknown impulse response is modeled as a zero-mean Gaussian process whose covariance (kernel) is given by the recently proposed stable spline kernel, which encodes information on regularity and exponential stability. To build robustness to outliers, we model the measurement noise as realizations of independent Laplacian random variables. The identification problem is cast in a Bayesian framework, and solved by a new Markov Chain Monte Carlo (MCMC) scheme. In particular, exploiting the representation of the Laplacian random variables as scale mixtures of Gaussians, we design a Gibbs sampler which quickly converges to the target distribution. Numerical simulations show a substantial improvement in the accuracy of the estimates over state-of-the-art kernel-based methods.