Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning
For researchers in representation learning, this work provides theoretical and empirical insights into how MRL induces interpretable, task-aligned structure in learned representations.
The paper proves that full-prefix Matryoshka Representation Learning (MRL) recovers ordered principal directions in linear settings and empirically shows that MRL yields a task-aligned privileged basis where coordinate magnitude reflects informativeness.
Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer-induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.