Dong Lu

Dong Lu, Zhiqiang Wang, Teng Wang et al.

Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art.

Dong Lu, Shanqin Chen, Yong-Tao Zhang

The weighted essentially non-oscillatory (WENO) schemes are a popular class of high order accurate numerical methods for solving hyperbolic partial differential equations (PDEs). The computational cost of such schemes increases significantly when the spatial dimensions of the PDEs are high, due to large number of spatial grid points and nonlinearity of high order accuracy WENO schemes. How to achieve fast computations by WENO methods for high spatial dimension PDEs is a challenging and important question. Recently, sparse-grid has become a major approximation tool for high dimensional problems. The open question is how to design WENO computations on sparse grids such that comparable high order accuracy of WENO schemes in smooth regions and essentially non-oscillatory stability in non-smooth regions of the solutions can still be achieved as that for computations on regular single grids? In this paper, we combine the third order finite difference WENO method with sparse-grid combination technique and solve high spatial dimension hyperbolic equations on sparse grids. WENO interpolation is proposed for the prolongation part in sparse grid combination techniques to deal with discontinuous solutions of hyperbolic equations. Numerical examples are presented to show that significant computational times are saved while both high order accuracy and stability of the WENO scheme are maintained for simulations on sparse grids.

Dong Lu, Yuanyuan Ruan, Dingkang Wang et al.

This paper investigates the Smith normal form equivalence problem for multivariate polynomial matrices. Using methods from matrix theory and polynomial ideal theory, we prove that Frost and Storey's 1978 conjecture holds for a broad class of matrices: such a matrix is equivalent to its Smith normal form if and only if its reduced minors of each order generate the unit ideal. Moreover, by extending the original matrix class via automorphisms of the polynomial ring, we show that our framework applies in a substantially more general setting.

Dong Lu, Tianyu Pang, Chao Du et al. · tsinghua

Backdoor attacks are commonly executed by contaminating training data, such that a trigger can activate predetermined harmful effects during the test phase. In this work, we present AnyDoor, a test-time backdoor attack against multimodal large language models (MLLMs), which involves injecting the backdoor into the textual modality using adversarial test images (sharing the same universal perturbation), without requiring access to or modification of the training data. AnyDoor employs similar techniques used in universal adversarial attacks, but distinguishes itself by its ability to decouple the timing of setup and activation of harmful effects. In our experiments, we validate the effectiveness of AnyDoor against popular MLLMs such as LLaVA-1.5, MiniGPT-4, InstructBLIP, and BLIP-2, as well as provide comprehensive ablation studies. Notably, because the backdoor is injected by a universal perturbation, AnyDoor can dynamically change its backdoor trigger prompts/harmful effects, exposing a new challenge for defending against backdoor attacks. Our project page is available at https://sail-sg.github.io/AnyDoor/.

Xiaolin Fan, Yan Wang, Yingying Zhang et al.

Automatic view positioning is crucial for cardiac computed tomography (CT) examinations, including disease diagnosis and surgical planning. However, it is highly challenging due to individual variability and large 3D search space. Existing work needs labor-intensive and time-consuming manual annotations to train view-specific models, which are limited to predicting only a fixed set of planes. However, in real clinical scenarios, the challenge of positioning semantic 2D slices with any orientation into varying coordinate space in arbitrary 3D volume remains unsolved. We thus introduce a novel framework, AVP-AP, the first to use Atlas Prompting for self-supervised Automatic View Positioning in the 3D CT volume. Specifically, this paper first proposes an atlas prompting method, which generates a 3D canonical atlas and trains a network to map slices into their corresponding positions in the atlas space via a self-supervised manner. Then, guided by atlas prompts corresponding to the given query images in a reference CT, we identify the coarse positions of slices in the target CT volume using rigid transformation between the 3D atlas and target CT volume, effectively reducing the search space. Finally, we refine the coarse positions by maximizing the similarity between the predicted slices and the query images in the feature space of a given foundation model. Our framework is flexible and efficient compared to other methods, outperforming other methods by 19.8% average structural similarity (SSIM) in arbitrary view positioning and achieving 9% SSIM in two-chamber view compared to four radiologists. Meanwhile, experiments on a public dataset validate our framework's generalizability.

Kunze Wang, Dong Lu, Soyeon Caren Han et al.

Online abusive language detection (ALD) has become a societal issue of increasing importance in recent years. Several previous works in online ALD focused on solving a single abusive language problem in a single domain, like Twitter, and have not been successfully transferable to the general ALD task or domain. In this paper, we introduce a new generic ALD framework, MACAS, which is capable of addressing several types of ALD tasks across different domains. Our generic framework covers multi-aspect abusive language embeddings that represent the target and content aspects of abusive language and applies a textual graph embedding that analyses the user's linguistic behaviour. Then, we propose and use the cross-attention gate flow mechanism to embrace multiple aspects of abusive language. Quantitative and qualitative evaluation results show that our ALD algorithm rivals or exceeds the six state-of-the-art ALD algorithms across seven ALD datasets covering multiple aspects of abusive language and different online community domains.