Peijia Qin

Peijia Qin, Ruiyi Zhang, Pengtao Xie

Parameter-efficient fine-tuning (PEFT) is a flexible and efficient method for adapting large language models (LLMs) to downstream tasks. Among these methods, weight-decomposed low-rank adaptation (DoRA) is a promising approach that decomposes weight matrices into magnitude and direction components to mimic full fine-tuning (FT) better. However, DoRA's simultaneous optimization of these components makes it over-expressive, increases the risk of overfitting, and creates a coupled updating pattern that limits its learning capacity. To address these issues, we propose Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), a novel PEFT method based on a bi-level optimization framework. BiDoRA fundamentally differs from DoRA by optimizing the magnitude and direction in two separate, asynchronous loops using distinct training and validation data splits. This decoupled optimization process effectively mitigates overfitting and allows for more flexible updates that align even more closely with FT. For instance, weight decomposition analysis shows BiDoRA achieves a magnitude-direction update correlation of $-8.042$, significantly closer to the FT ideal compared to $-1.784$ for DoRA. Evaluation of BiDoRA on diverse tasks spanning natural language understanding, generation, token classification, and extremely small biomedical datasets reveals that it consistently outperforms DoRA and a wide range of leading PEFT methods. This improvement is statistically significant, as demonstrated on the GLUE benchmark where BiDoRA surpasses DoRA with a p-value of $2.4\times10^{-4}$ in terms of the Wilcoxon signed-rank test. The code for BiDoRA is available at https://github.com/t2ance/BiDoRA.

Luke Bhan, Peijia Qin, Miroslav Krstic et al.

Predictor feedback designs are critical for delay-compensating controllers in nonlinear systems. However, these designs are limited in practical applications as predictors cannot be directly implemented, but require numerical approximation schemes, which become computationally prohibitive when system dynamics are expensive to compute. To address this challenge, we recast the predictor design as an operator learning problem, and learn the predictor mapping via a neural operator. We prove the existence of an arbitrarily accurate neural operator approximation of the predictor operator. Under the approximated predictor, we achieve semiglobal practical stability of the closed-loop nonlinear delay system. The estimate is semiglobal in a unique sense - one can enlarge the set of initial states as desired, though this increases the difficulty of training a neural operator, which appears practically in the stability estimate. Furthermore, our analysis holds for any black-box predictor satisfying the universal approximation error bound. We demonstrate the approach by controlling a 5-link robotic manipulator with different neural operator models, achieving significant speedups compared to classic predictor feedback schemes while maintaining closed-loop stability.