CSULoRA: Closest Safe Update Low-Rank Adaptation
This work is significant for researchers and practitioners deploying large language models, as it offers a method to mitigate safety risks introduced by fine-tuning with potentially unsafe data, which is an incremental improvement over existing methods.
This paper addresses the issue of safety degradation in large language models when fine-tuned with low-rank adaptation (LoRA) using unsafe data. The authors propose CSULoRA, a post-hoc method that corrects trained LoRA adapters by estimating a safety-aligned subspace and smoothly attenuating potentially unsafe update directions. In adversarial fine-tuning experiments, CSULoRA significantly reduces attack success rate while largely maintaining utility gains.
Low-rank adaptation has become a standard method for parameter-efficient fine-tuning of large language models, but even small amounts of unsafe or adversarial fine-tuning data can substantially weaken the safety behavior of aligned models. Existing safety-preserving LoRA methods often rely on hard interventions such as projection, pruning, thresholding, or additional training objectives. While these methods can suppress unsafe update directions, they may also remove task-relevant information or require extra tuning. We introduce CSULoRA, a post-hoc method for correcting trained LoRA adapters through closest safe update estimation. CSULoRA estimates a safety-aligned subspace from the weight displacement between a safety-aligned model and its corresponding base checkpoint. It then decomposes each LoRA update into fully aligned, partially aligned, and off-subspace components. Instead of discarding components outside the estimated safety subspace, CSULoRA solves a closed-form penalized minimum-change problem that preserves the fully aligned component while smoothly attenuating potentially unsafe directions according to their relative energy. In adversarial fine-tuning experiments, CSULoRA substantially reduces attack success rate while preserving most of the utility gains obtained from standard LoRA fine-tuning.