Output-Space Search: Targeting LLM Generations in a Frozen Encoder-Defined Output Space
This addresses the challenge of enhancing output diversity and targeted optimization for users of large language models, though it appears incremental as it builds on existing encoder and RL methods.
The paper tackles the problem of improving diversity and objective optimization in LLM outputs by introducing Output-Space Search (OS-Search), which transforms generation into endpoint search in a frozen encoder-defined output space, resulting in 3.1x higher diversity in stories and improved objective optimization in code while preserving validity.
We introduce Output-Space Search (OS-Search), which turns LLM generation into endpoint search. An outer loop selects a target z* in a frozen encoder-defined 3D output space Z, and a retrieval-grounded policy trained with sequence-level RL generates outputs whose coordinates land near z* under standard autoregressive decoding. This enables parallel sweeps and black-box optimization in Z without path-dependent token/program search. On stories, sweeping Z (text) yields 3.1x higher LLM-scored diversity than prompt-chaining. On code, Bayesian optimization over Z (code) improves an objective withheld from the controller under matched inference budgets while preserving validity.