Beyond Multiple Choice: Evaluating Steering Vectors for Adaptive Free-Form Summarization
This work addresses the understudied effectiveness of steering vectors in free-form generation tasks for researchers and practitioners in NLP, though it is incremental as it extends existing methods to a new domain.
The paper tackled the problem of evaluating steering vectors for controlling text properties in free-form summarization, finding that steering effectively controls properties like topical focus and sentiment but degrades text quality at high strengths, while combining steering with prompting offers the best trade-off.
Steering vectors are a lightweight method for controlling text properties by adding a learned bias to language model activations at inference time. So far, steering vectors have predominantly been evaluated in multiple-choice settings, while their effectiveness in free-form generation tasks remains understudied. Moving "Beyond Multiple Choice," we thoroughly evaluate the effectiveness of steering vectors in adaptively controlling topical focus, sentiment, toxicity, and readability in abstractive summaries of the NEWTS dataset. We find that steering effectively controls the targeted summary properties, but high steering strengths consistently degrade both intrinsic and extrinsic text quality. Compared to steering, prompting offers weaker control, while preserving text quality. Combining steering and prompting yields the strongest control over text properties and offers the most favorable efficacy-quality trade-off at moderate steering strengths. Our results underscore the practical trade-off between control strength and text quality preservation when applying steering vectors to free-form generation tasks.