Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2
This addresses a specific linguistic ambiguity in language models for NLP researchers, but it is incremental as it focuses on a narrow case in one model.
The study tackled the problem of language models' internal mechanisms for generating prepositional phrases, specifically distinguishing between instrumental and attributive complements in Gemma-2, and found that scaling a single attention head could shift the distribution from a 3:4 instrumental preference to 33% instrumental and 36% attributive.
Language Models, when generating prepositional phrases, must often decide for whether their complements functions as an instrumental adjunct (describing the verb adverbially) or an attributive modifier (enriching the noun adjectivally), yet the internal mechanisms that resolve this split decision remain poorly understood. In this study, we conduct a targeted investigation into Gemma-2 to uncover and control the generation of prepositional complements. We assemble a prompt suite containing with-headed prepositional phrases whose contexts equally accommodate either an instrumental or attributive continuation, revealing a strong preference for an instrumental reading at a ratio of 3:4. To pinpoint individual attention heads that favor instrumental over attributive complements, we project activations into the vocabulary space. By scaling the value vector of a single attention head, we can shift the distribution of functional roles of complements, attenuating instruments to 33% while elevating attributes to 36%.