Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs
This addresses the problem of knowledge degradation in RL-enhanced LLMs for researchers and practitioners, offering insights into procedural skill improvements rather than incremental gains.
The paper challenges the narrative that reinforcement learning (RL) degrades memorized knowledge in LLMs, showing that RL-enhanced models outperform base and supervised fine-tuned models on knowledge recall tasks, particularly for hierarchical knowledge like medical codes, with structured prompting reducing the performance gap from 24pp to 7pp on MedConceptsQA.
Reinforcement learning (RL) is often credited with improving language model reasoning and generalization at the expense of degrading memorized knowledge. We challenge this narrative by observing that RL-enhanced models consistently outperform their base and supervised fine-tuned (SFT) counterparts on pure knowledge recall tasks, particularly those requiring traversal of hierarchical, structured knowledge (e.g., medical codes). We hypothesize these gains stem not from newly acquired data, but from improved procedural skills in navigating and searching existing knowledge hierarchies within the model parameters. To support this hypothesis, we show that structured prompting, which explicitly guides SFTed models through hierarchical traversal, recovers most of the performance gap (reducing 24pp to 7pp on MedConceptsQA for DeepSeek-V3/R1). We further find that while prompting improves final-answer accuracy, RL-enhanced models retain superior ability to recall correct procedural paths on deep-retrieval tasks. Finally our layer-wise internal activation analysis reveals that while factual representations (e.g., activations for the statement "code 57.95 refers to urinary infection") maintain high cosine similarity between SFT and RL models, query representations (e.g., "what is code 57.95") diverge noticeably, indicating that RL primarily transforms how models traverse knowledge rather than the knowledge representation itself.