Yifeng Xu

Xin Feng, Yifeng Xu, Guangming Lu et al.

Effective image restoration with large-size corruptions, such as blind image inpainting, entails precise detection of corruption region masks which remains extremely challenging due to diverse shapes and patterns of corruptions. In this work, we present a novel method for automatic corruption detection, which allows for blind corruption restoration without known corruption masks. Specifically, we develop a hierarchical contrastive learning framework to detect corrupted regions by capturing the intrinsic semantic distinctions between corrupted and uncorrupted regions. In particular, our model detects the corrupted mask in a coarse-to-fine manner by first predicting a coarse mask by contrastive learning in low-resolution feature space and then refines the uncertain area of the mask by high-resolution contrastive learning. A specialized hierarchical interaction mechanism is designed to facilitate the knowledge propagation of contrastive learning in different scales, boosting the modeling performance substantially. The detected multi-scale corruption masks are then leveraged to guide the corruption restoration. Detecting corrupted regions by learning the contrastive distinctions rather than the semantic patterns of corruptions, our model has well generalization ability across different corruption patterns. Extensive experiments demonstrate following merits of our model: 1) the superior performance over other methods on both corruption detection and various image restoration tasks including blind inpainting and watermark removal, and 2) strong generalization across different corruption patterns such as graffiti, random noise or other image content. Codes and trained weights are available at https://github.com/xyfJASON/HCL .

Shaoming Duan, Chuanyi Liu, Peiyi Han et al. · microsoft-research

Non-independent and identically distributed (non-IID) data is a key challenge in federated learning (FL), which usually hampers the optimization convergence and the performance of FL. Existing data augmentation methods based on federated generative models or raw data sharing strategies for solving the non-IID problem still suffer from low performance, privacy protection concerns, and high communication overhead in decentralized tabular data. To tackle these challenges, we propose a federated tabular data augmentation method, named Fed-TDA. The core idea of Fed-TDA is to synthesize tabular data for data augmentation using some simple statistics (e.g., distributions of each column and global covariance). Specifically, we propose the multimodal distribution transformation and inverse cumulative distribution mapping respectively synthesize continuous and discrete columns in tabular data from a noise according to the pre-learned statistics. Furthermore, we theoretically analyze that our Fed-TDA not only preserves data privacy but also maintains the distribution of the original data and the correlation between columns. Through extensive experiments on five real-world tabular datasets, we demonstrate the superiority of Fed-TDA over the state-of-the-art in test performance and communication efficiency.

Tianhao Hu, Guanglian Li, Fengru Wang et al.

The reliable and accurate numerical approximation of the $p$-Laplacian is particularly challenging in the extreme regimes $p \to 1^{+}$ and $p \gg 1$, where the operator becomes either highly singular or strongly degenerate, often causing severe instability in standard numerical methods. To address these difficulties, we propose a novel deep learning based framework, termed the dual variational neural network, for $p$-Laplace problems. The approach is based on a mixed formulation and an $L^q$-based Helmholtz decomposition, which decouples the original problem into two convex subproblems: a linear Poisson problem for the irrotational component and an unconstrained minimization problem over divergence-free fields for the solenoidal component. Following the decomposition, we employ two neural networks using a gradient--curl representation to approximate the flux, and further establish an error analysis of the neural approximation. The analysis relies on fundamental vector inequalities together with tools from statistical learning theory. Numerical experiments demonstrate robust convergence of the proposed method in challenging settings, including the extreme cases $p \to 1^{+}$ and $p \gg 1$, as well as the $p(x)$-Laplace equation.

Qian Zhang, Long Chen, Yifeng Xu

In this paper an iterative minimization method is proposed to approximate the minimizer to the double-well energy functional arising in the phase-field theory. The method is based on a quadratic functional posed over a nonempty closed convex set and is shown to be unconditionally energy stable. By the minimization approach, we also derive an variant of the first-order scheme for the Allen-Cahn equation, which has been constructed in the context of Invariant Energy Quadratization, and prove its unconditional energy stability.

Guanglian Li, Yifeng Xu

In this work, we derive a reliable and efficient residual-typed error estimator for the finite element approximation of a 2d cathodic protection problem governed by a steady-state diffusion equation with a nonlinear boundary condition. We propose a standard adaptive finite element method involving the Dörfler marking and a minimal refinement without the interior node property. Furthermore, we establish the contraction property of this adaptive algorithm in terms of the sum of the energy error and the scaled estimator. This essentially allows for a quasi-optimal convergence rate in terms of the number of elements over the underlying triangulation. Numerical experiments are provided to confirm this quasi-optimality.

Yifeng Xu, Zhenliang He, Shiguang Shan et al.

Recently, large-scale diffusion models have made impressive progress in text-to-image (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at https://github.com/xyfJASON/ctrlora.

Xinyu Ma, Yifeng Xu, Yang Lin et al.

We introduce DRESS, a novel approach for generating stylized large language model (LLM) responses through representation editing. Existing methods like prompting and fine-tuning are either insufficient for complex style adaptation or computationally expensive, particularly in tasks like NPC creation or character role-playing. Our approach leverages the over-parameterized nature of LLMs to disentangle a style-relevant subspace within the model's representation space to conduct representation editing, ensuring a minimal impact on the original semantics. By applying adaptive editing strengths, we dynamically adjust the steering vectors in the style subspace to maintain both stylistic fidelity and semantic integrity. We develop two stylized QA benchmark datasets to validate the effectiveness of DRESS, and the results demonstrate significant improvements compared to baseline methods such as prompting and ITI. In short, DRESS is a lightweight, train-free solution for enhancing LLMs with flexible and effective style control, making it particularly useful for developing stylized conversational agents. Codes and benchmark datasets are available at https://github.com/ArthurLeoM/DRESS-LLM.

Yifeng Xu, Zhenliang He, Meina Kan et al.

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.

Bangti Jin, Jing Li, Yifeng Xu et al.

In this work, we develop an adaptive nonconforming finite element algorithm for the numerical approximation of phase-field parameterized topology optimization governed by the Stokes system. We employ the conforming linear finite element space to approximate the phase field, and the nonconforming linear finite elements (Crouzeix-Raviart elements) and piecewise constants to approximate the velocity field and the pressure field, respectively. We establish the convergence of the adaptive method, i.e., the sequence of minimizers contains a subsequence that converges to a solution of the first-order optimality system, and the associated subsequence of discrete pressure fields also converges. The analysis relies crucially on a new discrete compactness result of nonconforming linear finite elements over a sequence of adaptively generated meshes. We present numerical results for several examples to illustrate the performance of the algorithm, including a comparison with the uniform refinement strategy.

Bangti Jin, Yifeng Xu

In this work we propose and analyze a numerical method for electrical impedance tomography of recovering a piecewise constant conductivity from boundary voltage measurements. It is based on standard Tikhonov regularization with a Modica-Mortola penalty functional and adaptive mesh refinement using suitable a posteriori error estimators of residual type that involve the state, adjoint and variational inequality in the necessary optimality condition and a separate marking strategy. We prove the convergence of the adaptive algorithm in the following sense: the sequence of discrete solutions contains a subsequence convergent to a solution of the continuous necessary optimality system. Several numerical examples are presented to illustrate the convergence behavior of the algorithm.

Bangti Jin, Yifeng Xu, Jun Zou

In this work we develop and analyze an adaptive finite element method for efficiently solving electrical impedance tomography -- a severely ill-posed nonlinear inverse problem for recovering the conductivity from boundary voltage measurements. The reconstruction technique is based on Tikhonov regularization with a Sobolev smoothness penalty and discretizing the forward model using continuous piecewise linear finite elements. We derive an adaptive finite element algorithm with an a posteriori error estimator involving the concerned state and adjoint variables and the recovered conductivity. The convergence of the algorithm is established, in the sense that the sequence of discrete solutions contains a convergent subsequence to a solution of the optimality system for the continuous formulation. Numerical results are presented to verify the convergence and efficiency of the algorithm.