Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs
This work addresses the need for fine-grained control over text attributes like conciseness and emotion in LLM generation, which is important for applications in writing and chatbots, but it is incremental as it builds on existing methods without introducing a new paradigm.
The paper tackles the problem of controlling attribute intensity in text generation using large language models (LLMs), proposing metrics and an evaluation framework to assess smooth control, and finds that training-free methods like prompting and representation modification achieve measurable improvements in range, calibration, and consistency across five attributes.
Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such \emph{smooth control} of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we propose an effective evaluation framework leveraging the Elo rating system and GPT4, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on $5$ different attributes with various models. Our code and dataset can be obtained from \url{https://github.com/ShangDataLab/Smooth-Control}.