Zhengqi Liu

Jie Gui, Tuo Chen, Minjing Dong et al.

Recently, masked image modeling (MIM), which learns visual representations by reconstructing the masked patches of an image, has dominated self-supervised learning in computer vision. However, the pre-training of MIM always takes massive time due to the large-scale data and large-size backbones. We mainly attribute it to the random patch masking in previous MIM works, which fails to leverage the crucial semantic information for effective visual representation learning. To tackle this issue, we propose the Frequency \& Attention-driven Masking and Throwing Strategy (FAMT), which can extract semantic patches and reduce the number of training patches to boost model performance and training efficiency simultaneously. Specifically, FAMT utilizes the self-attention mechanism to extract semantic information from the image for masking during training in an unsupervised manner. However, attention alone could sometimes focus on inappropriate areas regarding the semantic information. Thus, we are motivated to incorporate the information from the frequency domain into the self-attention mechanism to derive the sampling weights for masking, which captures semantic patches for visual representation learning. Furthermore, we introduce a patch throwing strategy based on the derived sampling weights to reduce the training cost. FAMT can be seamlessly integrated as a plug-and-play module and surpasses previous works, \emph{e.g.} reducing the training phase time by nearly $50\%$ and improving the linear probing accuracy of MAE by $1.3\% \sim 3.9\%$ across various datasets, including CIFAR-10/100, Tiny ImageNet, and ImageNet-1K. FAMT also demonstrates superior performance in downstream detection and segmentation tasks.

Sanghyun Lee, Zhengqi Liu, Jonathan Valyou et al.

Ensemble Kalman Filter (EnKF) based data assimilation algorithms synthesize predictive numerical forecast models with accumulated data as time evolves and account for model uncertainty and noisy measurements. The computational cost of these algorithms can be expensive, in particular for highly dimensional dynamical systems. Often, EnKF based algorithms have traded accuracy for reduced computational cost. In this paper, we present a novel parallelizable Conjugate Gradient-based Ensemble Kalman Filter (CGD-EnKF) algorithm that maintains comparable computational cost to efficient algorithms while realizing better state estimation accuracy in select cases. Here, we established the new approach by reformulating a matrix inverse calculation with a classical Conjugate Gradient (CGD) method. In addition, we discuss the upper error bound under CGD, error convergence to the classical EnKF result, and the computational complexity of the algorithm. We also showcase the CGD-EnKF-Reduced algorithm that is shown to be further computationally efficient for highly dimensional dynamical systems under small ensemble formulation. Numerical examples demonstrate the performance of our proposed algorithms and analytical properties, highlighting their comparability and advantages with respect to some benchmark EnKF algorithms.

Zhengqi Liu, Shuhao Cao, Yuwen Li et al.

The training process of neural networks usually optimize weights and bias parameters of linear transformations, while nonlinear activation functions are pre-specified and fixed. This work develops a systematic approach to constructing matrix-valued activation functions whose entries are generalized from ReLU. The activation is based on matrix-vector multiplications using only scalar multiplications and comparisons. The proposed activation functions depend on parameters that are trained along with the weights and bias vectors. Neural networks based on this approach are simple and efficient and are shown to be robust in numerical experiments.