Does Representation Matter? Exploring Intermediate Layers in Large Language Models
This work addresses the problem of understanding and optimizing LLM representations for researchers and practitioners, but it is incremental as it applies existing metrics to new contexts.
The paper investigates the quality of intermediate representations in large language models (LLMs) like Transformers and State Space Models, finding that these layers often provide more informative representations for downstream tasks than final layers, with observations including a bimodal pattern in entropy.
Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we investigate the quality of intermediate representations in various LLM architectures, including Transformers and State Space Models (SSMs). We find that intermediate layers often yield more informative representations for downstream tasks than the final layers. To measure the representation quality, we adapt and apply a suite of metrics - such as prompt entropy, curvature, and augmentation-invariance - originally proposed in other contexts. Our empirical study reveals significant architectural differences, how representations evolve throughout training, and how factors like input randomness and prompt length affect each layer. Notably, we observe a bimodal pattern in the entropy of some intermediate layers and consider potential explanations tied to training data. Overall, our results illuminate the internal mechanics of LLMs and guide strategies for architectural optimization and training.