Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges
This work provides a tool for researchers and practitioners to better understand and refine prompts for language models, though it is incremental as it builds on existing beam search algorithms.
The paper tackles challenges in prompting large language models by introducing an interactive visual method to analyze beam search trees, showing quantitatively that this exposure helps address data-, model-, linguistic, and socio-linguistic issues.
The growing popularity of generative language models has amplified interest in interactive methods to guide model outputs. Prompt refinement is considered one of the most effective means to influence output among these methods. We identify several challenges associated with prompting large language models, categorized into data- and model-specific, linguistic, and socio-linguistic challenges. A comprehensive examination of model outputs, including runner-up candidates and their corresponding probabilities, is needed to address these issues. The beam search tree, the prevalent algorithm to sample model outputs, can inherently supply this information. Consequently, we introduce an interactive visual method for investigating the beam search tree, facilitating analysis of the decisions made by the model during generation. We quantitatively show the value of exposing the beam search tree and present five detailed analysis scenarios addressing the identified challenges. Our methodology validates existing results and offers additional insights.