Exploring Concreteness Through a Figurative Lens
For NLP researchers, this work provides mechanistic insights into how LLMs handle context-dependent concreteness shifts, with practical applications for figurative language processing.
The paper investigates how LLMs internally represent concreteness in figurative language, finding that models separate literal and figurative usage in early layers and compress concreteness into a one-dimensional direction in mid-to-late layers, which enables efficient figurative-language classification and training-free steering of generation.
Static concreteness ratings are widely used in NLP, yet a word's concreteness can shift with context, especially in figurative language such as metaphor, where common concrete nouns can take abstract interpretations. While such shifts are evident from context, it remains unclear how LLMs understand concreteness internally. We conduct a layer-wise and geometric analysis of LLM hidden representations across four model families, examining how models distinguish literal vs figurative uses of the same noun and how concreteness is organized in representation space. We find that LLMs separate literal and figurative usage in early layers, and that mid-to-late layers compress concreteness into a one-dimensional direction that is consistent across models. Finally, we show that this geometric structure is practically useful: a single concreteness direction supports efficient figurative-language classification and enables training-free steering of generation toward more literal or more figurative rewrites.