Xiao Lin

Wenpeng Zhang, Xiao Lin, Peilin Zhao

Factorization Machine (FM) is a supervised learning approach with a powerful capability of feature engineering. It yields state-of-the-art performance in various batch learning tasks where all the training data is made available prior to the training. However, in real-world applications where the data arrives sequentially in a streaming manner, the high cost of re-training with batch learning algorithms has posed formidable challenges in the online learning scenario. The initial challenge is that no prior formulations of FM could fulfill the requirements in Online Convex Optimization (OCO) -- the paramount framework for online learning algorithm design. To address the aforementioned challenge, we invent a new convexification scheme leading to a Compact Convexified FM (CCFM) that seamlessly meets the requirements in OCO. However for learning Compact Convexified FM (CCFM) in the online learning setting, most existing algorithms suffer from expensive projection operations. To address this subsequent challenge, we follow the general projection-free algorithmic framework of Online Conditional Gradient and propose an Online Compact Convex Factorization Machine (OCCFM) algorithm that eschews the projection operation with efficient linear optimization steps. In support of the proposed OCCFM in terms of its theoretical foundation, we prove that the developed algorithm achieves a sub-linear regret bound. To evaluate the empirical performance of OCCFM, we conduct extensive experiments on 6 real-world datasets for online recommendation and binary classification tasks. The experimental results show that OCCFM outperforms the state-of-art online learning algorithms.

Chao Chen, Xiao Lin, Gabriel Terejanu

Long Short-Term Memory networks trained with gradient descent and back-propagation have received great success in various applications. However, point estimation of the weights of the networks is prone to over-fitting problems and lacks important uncertainty information associated with the estimation. However, exact Bayesian neural network methods are intractable and non-applicable for real-world applications. In this study, we propose an approximate estimation of the weights uncertainty using Ensemble Kalman Filter, which is easily scalable to a large number of weights. Furthermore, we optimize the covariance of the noise distribution in the ensemble update step using maximum likelihood estimation. To assess the proposed algorithm, we apply it to outlier detection in five real-world events retrieved from the Twitter platform.

Xiao Lin, Gabriel Terejanu

Real-time nonlinear Bayesian filtering algorithms are overwhelmed by data volume, velocity and increasing complexity of computational models. In this paper, we propose a novel ensemble based nonlinear Bayesian filtering approach which only requires a small number of simulations and can be applied to high-dimensional systems in the presence of intractable likelihood functions. The proposed approach uses linear latent projections to estimate the joint probability distribution between states, parameters, and observables using a mixture of Gaussian components generated by the reconstruction error for each ensemble member. Since it leverages the computational machinery behind linear latent variable models, it can achieve fast implementations without the need to compute high-dimensional sample covariance matrices. The performance of the proposed approach is compared with the performance of ensemble Kalman filter on a high-dimensional Lorenz nonlinear dynamical system.

Xiao Lin, Asif Chowdhury, Xiaofan Wang et al.

Since the cost of installing and maintaining sensors is usually high, sensor locations are always strategically selected. For those aiming at inferring certain quantities of interest (QoI), it is desirable to explore the dependency between sensor measurements and QoI. One of the most popular metric for the dependency is mutual information which naturally measures how much information about one variable can be obtained given the other. However, computing mutual information is always challenging, and the result is unreliable in high dimension. In this paper, we propose an approach to find an approximate lower bound of mutual information and compute it in a lower dimension. Then, sensors are placed where highest mutual information (lower bound) is achieved and QoI is inferred via Bayes rule given sensor measurements. In addition, Bayesian optimization is introduced to provide a continuous mutual information surface over the domain and thus reduce the number of evaluations. A chemical release accident is simulated where multiple sensors are placed to locate the source of the release. The result shows that the proposed approach is both effective and efficient in inferring QoI.