Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures
For researchers studying cross-architecture concept alignment in LLMs, this work provides a training-free diagnostic to detect concept-specific convergence and architectural outliers, though it is an incremental methodological contribution.
The paper investigates whether different LLM architectures encode high-level concepts in structurally compatible ways, finding a dissociation between moderate geometric convergence and near-perfect functional transfer across multiple concept domains. The proposed CKA_Delta diagnostic achieves significant discrimination where standard CKA fails, with p <= 0.017 for five of six concept domains.
Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA_Delta), a training-free diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity -- achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p <= 0.017 geometric discrimination and safety as a converging-functional trend, p = 0.08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B--70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional >=70B models. We position CKA_Delta as a practical regime classifier and architectural outlier detector (Gemma: d = 1.08, AUC = 0.79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring.