Mihaly Petreczky

Ion Victor Gosea, Mihaly Petreczky, Athanasios C. Antoulas et al.

We propose a model order reduction approach for balanced truncation of linear switched systems. Such systems switch among a finite number of linear subsystems or modes. We compute pairs of controllability and observability Gramians corresponding to each active discrete mode by solving systems of coupled Lyapunov equations. Depending on the type, each such Gramian corresponds to the energy associated to all possible switching scenarios that start or, respectively end, in a particular operational mode. In order to guarantee that hard to control and hard to observe states are simultaneously eliminated, we construct a transformed system, whose Gramians are equal and diagonal. Then, by truncation, directly construct reduced order models. One can show that these models preserve some properties of the original model, such as stability and that it is possible to obtain error bounds relating the observed output, the control input and the entries of the diagonal Gramians.

Ion Victor Gosea, Mihaly Petreczky, Athanasios C. Antoulas

The Loewner framework for model reduction is extended to the class of linear switched systems. One advantage of this framework is that it introduces a trade-off between accuracy and complexity. Moreover, through this procedure, one can derive state-space models directly from data which is related to the input-output behavior of the original system. Hence, another advantage of the framework is that it does not require the initial system matrices. More exactly, the data used in this framework consists in frequency domain samples of input-output mappings of the original system. The definition of generalized transfer functions for linear switched systems resembles the one for bilinear systems. A key role is played by the coupling matrices, which ensure the transition from one active mode to another.

Mihaly Petreczky, Laurent Bako

The paper formulates the concept of persistence of excitation for discrete-time linear switched systems, and provides sufficient conditions for an input signal to be persistently exciting. Persistence of excitation is formulated as a property of the input signal, and it is not tied to any specific identification algorithm. The results of the paper rely on realization theory and on the notion of Markov-parameters for linear switched systems.

Deividas Eringis, John Leth, Zheng-Hua Tan et al.

In this paper we derive a Probably Approxilmately Correct(PAC)-Bayesian error bound for linear time-invariant (LTI) stochastic dynamical systems with inputs. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. In particular, with the bound derived in this paper relates future average prediction errors with the prediction error generated by the model on the data used for learning. In turn, this allows us to provide finite-sample error bounds for a wide class of learning/system identification algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-Bayesian bounds for RNNs.

Deividas Eringis, John Leth, Zheng-Hua Tan et al.

In this paper we derive a PAC-Bayesian-Like error bound for a class of stochastic dynamical systems with inputs, namely, for linear time-invariant stochastic state-space models (stochastic LTI systems for short). This class of systems is widely used in control engineering and econometrics, in particular, they represent a special case of recurrent neural networks. In this paper we 1) formalize the learning problem for stochastic LTI systems with inputs, 2) derive a PAC-Bayesian-Like error bound for such systems, 3) discuss various consequences of this error bound.

Manas Mejari, Mihaly Petreczky

In this paper, we present a realization and an identification algorithm for stochastic Linear Parameter-Varying State-Space Affine (LPV-SSA) representations. The proposed realization algorithm combines the deterministic LPV input output to LPV state-space realization scheme based on correlation analysis with a stochastic covariance realization algorithm. Based on this realization algorithm, a computationally efficient and statistically consistent identification algorithm is proposed to estimate the LPV model matrices, which are computed from the empirical covariance matrices of outputs, inputs and scheduling signal observations. The effectiveness of the proposed algorithm is shown via a numerical case study.

Mert Bastug, Laurentiu Hetel, Mihaly Petreczky

Two approaches to moment matching based model reduction of aperiodically sampled data systems are given. The term "aperiodic sampling" is used in the paper to indicate that the time between two consecutive sampling instants can take its value from a pre-specified finite set of allowed sampling intervals. Such systems can be represented by discrete-time linear switched (LS) state space (SS) models. One of the approaches investigated in the paper is to apply model reduction by moment matching on the linear time-invariant (LTI) plant model, then compare the responses of the LS SS models acquired from the original and reduced order LTI plants. The second approach is to apply a moment matching based model reduction method on the LS SS model acquired from the original LTI plant, and then compare the responses of the original and reduced LS SS models. It is proven that for both methods, as long as the original LTI plant is stable, the resulting reduced order LS SS model of the sampled data system is quadratically stable. The results from two approaches are compared with numerical examples.

Martin Gonzalez, Thibault Defourneau, Hatem Hajri et al.

In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-output map to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems. These results represent the first steps towards realization theory of recurrent neural ODE architectures, which is is expected be useful for model reduction and learning algorithm analysis of recurrent neural ODEs.

Martin Gonzalez, Hatem Hajri, Loic Cantat et al.

We investigate the problems and challenges of evaluating the robustness of Differential Equation-based (DE) networks against synthetic distribution shifts. We propose a novel and simple accuracy metric which can be used to evaluate intrinsic robustness and to validate dataset corruption simulators. We also propose methodology recommendations, destined for evaluating the many faces of neural DEs' robustness and for comparing them with their discrete counterparts rigorously. We then use this criteria to evaluate a cheap data augmentation technique as a reliable way for demonstrating the natural robustness of neural ODEs against simulated image corruptions across multiple datasets.

Deividas Eringis, John Leth, Zheng-Hua Tan et al.

In this paper, we derive a PAC-Bayes bound on the generalisation gap, in a supervised time-series setting for a special class of discrete-time non-linear dynamical systems. This class includes stable recurrent neural networks (RNN), and the motivation for this work was its application to RNNs. In order to achieve the results, we impose some stability constraints, on the allowed models. Here, stability is understood in the sense of dynamical systems. For RNNs, these stability conditions can be expressed in terms of conditions on the weights. We assume the processes involved are essentially bounded and the loss functions are Lipschitz. The proposed bound on the generalisation gap depends on the mixing coefficient of the data distribution, and the essential supremum of the data. Furthermore, the bound converges to zero as the dataset size increases. In this paper, we 1) formalize the learning problem, 2) derive a PAC-Bayesian error bound for such systems, 3) discuss various consequences of this error bound, and 4) show an illustrative example, with discussions on computing the proposed bound. Unlike other available bounds the derived bound holds for non i.i.d. data (time-series) and it does not grow with the number of steps of the RNN.

Daniel Racz, Mihaly Petreczky, Balint Daroczy

We derive a finite-sample probabilistic bound on the parameter estimation error of a system identification algorithm for Linear Switched Systems. The algorithm estimates Markov parameters from a single trajectory and applies a variant of the Ho-Kalman algorithm to recover the system matrices. Our bound guarantees statistical consistency under the assumption that the true system exhibits quadratic stability. The proof leverages the theory of weakly dependent processes. To the best of our knowledge, this is the first finite-sample bound for this algorithm in the single-trajectory setting.

Daniel Racz, Martin Gonzalez, Mihaly Petreczky et al.

One of the main theoretical challenges in learning dynamical systems from data is providing upper bounds on the generalization error, that is, the difference between the expected prediction error and the empirical prediction error measured on some finite sample. In machine learning, a popular class of such bounds are the so-called Probably Approximately Correct (PAC) bounds. In this paper, we derive a PAC bound for stable continuous-time linear parameter-varying (LPV) systems. Our bound depends on the H2 norm of the chosen class of the LPV systems, but does not depend on the time interval for which the signals are considered.

Deividas Eringis, John Leth, Zheng-Hua Tan et al.

In this short article, we showcase the derivation of the optimal (minimum error variance) estimator, when one part of the stochastic LTI system output is not measured but is able to be predicted from the measured system outputs. Similar derivations have been done before but not using state-space representation.

Deividas Eringis, John Leth, Zheng-Hua Tan et al.

In this paper we derive a PAC-Bayesian error bound for autonomous stochastic LTI state-space models. The motivation for deriving such error bounds is that they will allow deriving similar error bounds for more general dynamical systems, including recurrent neural networks. In turn, PACBayesian error bounds are known to be useful for analyzing machine learning algorithms and for deriving new ones.

Vera Shalaeva, Alireza Fakhrizadeh Esfahani, Pascal Germain et al.

In this paper, we improve the PAC-Bayesian error bound for linear regression derived in Germain et al. [10]. The improvements are twofold. First, the proposed error bound is tighter, and converges to the generalization loss with a well-chosen temperature parameter. Second, the error bound also holds for training data that are not independently sampled. In particular, the error bound applies to certain time series generated by well-known classes of dynamical models, such as ARX models.