DeLTa: A Decoding Strategy based on Logit Trajectory Prediction Improves Factuality and Reasoning Ability
This addresses reliability issues in LLMs for real-world applications, offering an incremental improvement without architectural changes.
The paper tackles the problem of unreliable content generation in Large Language Models (LLMs), such as factual inaccuracies and logical reasoning deficiencies, by proposing a novel decoding strategy called DeLTa that adjusts next-token probabilities based on logit trajectory analysis, resulting in improvements of up to 4.9% on TruthfulQA, 8.1% on StrategyQA, and 7.3% on GSM8K.
Large Language Models (LLMs) are increasingly being used in real-world applications. However, concerns about the reliability of the content they generate persist, as it frequently deviates from factual correctness or exhibits deficiencies in logical reasoning. This paper proposes a novel decoding strategy aimed at enhancing both factual accuracy and inferential reasoning without requiring any modifications to the architecture or pre-trained parameters of LLMs. Our approach adjusts next-token probabilities by analyzing the trajectory of logits from lower to higher layers in Transformers and applying linear regression. We find that this Decoding by Logit Trajectory-based approach (DeLTa) effectively reinforces factuality and reasoning while mitigating incorrect generation. Experiments on TruthfulQA demonstrate that DeLTa attains up to a 4.9% improvement over the baseline. Furthermore, it enhances performance by up to 8.1% on StrategyQA and 7.3% on GSM8K, both of which demand strong reasoning capabilities.