Refine and Purify: Orthogonal Basis Optimization with Null-Space Denoising for Conditional Representation Learning
This work addresses limitations in conditional representation learning for customized AI tasks, representing an incremental improvement over existing methods.
The paper tackles the problem of sensitivity to subspace basis and inter-subspace interference in conditional representation learning by proposing OD-CRL, which integrates Adaptive Orthogonal Basis Optimization and Null-Space Denoising Projection, achieving new state-of-the-art performance across tasks like customized clustering, classification, and retrieval.
Conditional representation learning aims to extract criterion-specific features for customized tasks. Recent studies project universal features onto the conditional feature subspace spanned by an LLM-generated text basis to obtain conditional representations. However, such methods face two key limitations: sensitivity to subspace basis and vulnerability to inter-subspace interference. To address these challenges, we propose OD-CRL, a novel framework integrating Adaptive Orthogonal Basis Optimization (AOBO) and Null-Space Denoising Projection (NSDP). Specifically, AOBO constructs orthogonal semantic bases via singular value decomposition with a curvature-based truncation. NSDP suppresses non-target semantic interference by projecting embeddings onto the null space of irrelevant subspaces. Extensive experiments conducted across customized clustering, customized classification, and customized retrieval tasks demonstrate that OD-CRL achieves a new state-of-the-art performance with superior generalization.