Zuogong Yue

Zuogong Yue, Johan Thunberg, Wei Pan et al.

Performing multiple experiments is common when learning internal mechanisms of complex systems. These experiments can include perturbations to parameters or external disturbances. A challenging problem is to efficiently incorporate all collected data simultaneously to infer the underlying dynamic network. This paper addresses the reconstruction of dynamic networks from heterogeneous datasets under the assumption that underlying networks share the same Boolean structure across all experiments. Parametric models for dynamical structure functions are derived to describe causal interactions between measured variables. Multiple datasets are integrated into one regression problem with additional demands of group sparsity to assure network sparsity and structure consistency. To acquire structured group sparsity, we propose a sampling-based method, together with extended versions of l1 methods and sparse Bayesian learning. The performance of the proposed methods is benchmarked in numerical simulation. In summary, this paper presents efficient methods on network reconstruction from multiple experiments, and reveals practical experience that could guide applications.

Zuogong Yue, Johan Thunberg, Lennart Ljung et al.

Dynamic network reconstruction has been shown to be challenging due to the requirements on sparse network structures and network identifiability. The direct parametric method (e.g., using ARX models) requires a large amount of parameters in model selection. Amongst the parametric models, only a restricted class can easily be used to address network sparsity without rendering the optimization problem intractable. To overcome these problems, this paper presents a state-space-based method, which significantly reduces the number of unknown parameters in model selection. Furthermore, we avoid various difficulties arising in gradient computation by using the Expectation Minimization (EM) algorithm instead. To enhance network sparsity, the prior distribution is constructed by using the Sparse Bayesian Learning (SBL) approach in the M-step. To solve the SBL problem, another EM algorithm is embedded, where we impose conditions on network identifiability in each iteration. In a sum, this paper provides a solution to reconstruct dynamic networks that avoids the difficulties inherent to gradient computation and simplifies the model selection.

Zuogong Yue, Johan Thunberg, Lennart Ljung et al.

This paper addresses identification of sparse linear and noise-driven continuous-time state-space systems, i.e., the right-hand sides in the dynamical equations depend only on a subset of the states. The key assumption in this study, is that the sample rate is not high enough to directly infer the continuous time system from the data. This assumption is relevant in applications where sampling is expensive or requires human intervention (e.g., biomedicine applications). We propose an iterative optimization scheme with $l_1$-regularization, where the search directions are restricted those that decrease prediction error in each iteration. We provide numerical examples illustrating the proposed method; the method outperforms the least squares estimation for large noise.

Xiaoyu Liu, Beitong Zhou, Zuogong Yue et al.

Recently, the usage of Contrastive Representation Learning (CRL) as a pre-training technique improves the performance of learning with noisy labels (LNL) methods. However, instead of pre-training, when trivially combining CRL loss with LNL methods as an end-to-end framework, the empirical experiments show severe degeneration of the performance. We verify through experiments that this issue is caused by optimization conflicts of losses and propose an end-to-end \textbf{PLReMix} framework by introducing a Pseudo-Label Relaxed (PLR) contrastive loss. This PLR loss constructs a reliable negative set of each sample by filtering out its inappropriate negative pairs, alleviating the loss conflicts by trivially combining these losses. The proposed PLR loss is pluggable and we have integrated it into other LNL methods, observing their improved performance. Furthermore, a two-dimensional Gaussian Mixture Model is adopted to distinguish clean and noisy samples by leveraging semantic information and model outputs simultaneously. Experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed method. Code is available at \url{https://github.com/lxysl/PLReMix}.

Zuogong Yue, Victor Solo

We develop hard clustering based on likelihood rather than distance and prove convergence. We also provide simulations and real data examples.

Zuogong Yue, Xinyi Wang, Victor Solo

Clustering of time series based on their underlying dynamics is keeping attracting researchers due to its impacts on assisting complex system modelling. Most current time series clustering methods handle only scalar time series, treat them as white noise, or rely on domain knowledge for high-quality feature construction, where the autocorrelation pattern/feature is mostly ignored. Instead of relying on heuristic feature/metric construction, the system identification approach allows treating vector time series clustering by explicitly considering their underlying autoregressive dynamics. We first derive a clustering algorithm based on a mixture autoregressive model. Unfortunately it turns out to have significant computational problems. We then derive a `small-noise' limiting version of the algorithm, which we call k-LMVAR (Limiting Mixture Vector AutoRegression), that is computationally manageable. We develop an associated BIC criterion for choosing the number of clusters and model order. The algorithm performs very well in comparative simulations and also scales well computationally.

Ye Yuan, Jun Liu, Dou Jin et al.

Traditional machine learning relies on a centralized data pipeline, i.e., data are provided to a central server for model training. In many applications, however, data are inherently fragmented. Such a decentralized nature of these databases presents the biggest challenge for collaboration: sending all decentralized datasets to a central server raises serious privacy concerns. Although there has been a joint effort in tackling such a critical issue by proposing privacy-preserving machine learning frameworks, such as federated learning, most state-of-the-art frameworks are built still in a centralized way, in which a central client is needed for collecting and distributing model information (instead of data itself) from every other client, leading to high communication pressure and high vulnerability when there exists a failure at or attack on the central client. Here we propose a principled decentralized federated learning algorithm (DeceFL), which does not require a central client and relies only on local information transmission between clients and their neighbors, representing a fully decentralized learning framework. It has been further proven that every client reaches the global minimum with zero performance gap and achieves the same convergence rate $O(1/T)$ (where $T$ is the number of iterations in gradient descent) as centralized federated learning when the loss function is smooth and strongly convex. Finally, the proposed algorithm has been applied to a number of applications to illustrate its effectiveness for both convex and nonconvex loss functions, demonstrating its applicability to a wide range of real-world medical and industrial applications.