Cross-Model Semantics in Representation Learning
This work addresses the issue of representational transferability for researchers and practitioners in machine learning, focusing on model distillation and modular learning, but it is incremental as it builds on prior studies on structured transformations.
The paper tackled the problem of internal representations in deep networks being sensitive to architecture-specific choices, and found that structural regularities like linear shaping operators and corrective paths improve representational alignment and stability across different architectures, as demonstrated through theoretical insights and controlled transfer experiments.
The internal representations learned by deep networks are often sensitive to architecture-specific choices, raising questions about the stability, alignment, and transferability of learned structure across models. In this paper, we investigate how structural constraints--such as linear shaping operators and corrective paths--affect the compatibility of internal representations across different architectures. Building on the insights from prior studies on structured transformations and convergence, we develop a framework for measuring and analyzing representational alignment across networks with distinct but related architectural priors. Through a combination of theoretical insights, empirical probes, and controlled transfer experiments, we demonstrate that structural regularities induce representational geometry that is more stable under architectural variation. This suggests that certain forms of inductive bias not only support generalization within a model, but also improve the interoperability of learned features across models. We conclude with a discussion on the implications of representational transferability for model distillation, modular learning, and the principled design of robust learning systems.