CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions
This provides incremental insights into transformer interpretability for researchers, complementing existing methods with a novel spectral analysis approach.
The paper tackles the problem of understanding transformer layer functions in large language models by introducing CAST, a probe-free framework that analyzes transformation matrices and spectral metrics, revealing distinct behaviors between encoder-only and decoder-only models, such as compression-expansion cycles in decoders and consistent high-rank processing in encoders.
Large language models have achieved remarkable success but remain largely black boxes with poorly understood internal mechanisms. To address this limitation, many researchers have proposed various interpretability methods including mechanistic analysis, probing classifiers, and activation visualization, each providing valuable insights from different perspectives. Building upon this rich landscape of complementary approaches, we introduce CAST (Compositional Analysis via Spectral Tracking), a probe-free framework that contributes a novel perspective by analyzing transformer layer functions through direct transformation matrix estimation and comprehensive spectral analysis. CAST offers complementary insights to existing methods by estimating the realized transformation matrices for each layer using Moore-Penrose pseudoinverse and applying spectral analysis with six interpretable metrics characterizing layer behavior. Our analysis reveals distinct behaviors between encoder-only and decoder-only models, with decoder models exhibiting compression-expansion cycles while encoder models maintain consistent high-rank processing. Kernel analysis further demonstrates functional relationship patterns between layers, with CKA similarity matrices clearly partitioning layers into three phases: feature extraction, compression, and specialization.