Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
This work addresses a bottleneck in adapter merging for fine-tuning large models, but it is incremental as it re-examines prior assumptions without achieving new gains.
The paper tackled the problem of interference when merging multiple LoRA modules for compositional adaptation by proposing Orthogonal Monte Carlo Dropout to enforce orthogonality without extra time complexity, but found that orthogonality alone does not lead to semantic disentanglement, suggesting it may be insufficient for true compositionality.
We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity. Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style. When multiple LoRA modules are merged, for example to generate an object in a particular style, their outputs (semantic vectors) may interfere with each other. Our method guarantees that merged LoRA modules remain orthogonal and thus free from direct interference. However, empirical analysis reveals that such orthogonality does not lead to the semantic disentanglement highlighted in prior work on compositional adaptation. This finding suggests that inter-LoRA orthogonality alone may be insufficient for achieving true semantic compositionality, prompting a re-examination of its role in adapter merging.