DORA Explorer: Improving the Exploration Ability of LLMs Without Training
For LLM agents in environments requiring exploration, DORA provides a simple, training-free method to improve diversity and performance without modifying the model.
LLMs for sequential decision-making struggle with exploration, leading to sub-optimal solutions. DORA Explorer, a training-free framework, improves exploration by generating diverse action candidates and selecting them via token log-probabilities, achieving UCB-competitive performance on Multi-Armed Bandit and improving Qwen2.5-7B's TextWorld performance from 29.2% to 45.5%.
Despite the rapid progress, LLMs for sequential decision-making (i.e., LLM agents) still struggle to produce diverse outputs. This leads to insufficient exploration, convergence to sub-optimal solutions, and becoming stuck in loops. Such limitations can be problematic in environments that require active exploration to gather information and make decisions. Sampling methods such as temperature scaling introduce token-level randomness but fail to produce enough diversity at the sequence level. We analyze LLM exploration in the classic Multi-Armed Bandit (MAB) setting and the Text Adventure Learning Environment Suite (TALES). We find that current decoding strategies and prompting methods like Chain-of-Thought and Tree-of-Thought are insufficient for robust exploration. To address this, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free framework for improving exploration in LLM agents. DORA generates diverse action candidates, scores them using token log-probabilities, and selects actions using a tunable exploration parameter. DORA achieves UCB-competitive performance on MAB and consistent gains across TALES, e.g., improving Qwen2.5-7B's performance from 29.2% to 45.5% in TextWorld. Our project is available at: https://dora-explore.github.io/.