Soft $Q(λ)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
For researchers in reinforcement learning, this provides a theoretical framework for multi-step off-policy learning with entropy regularization, but it is incremental as it combines existing ideas without empirical validation.
The paper extends soft Q-learning to multi-step off-policy settings by introducing a Soft Tree Backup operator and unifying it into Soft Q(λ), an eligibility trace framework for entropy-regularized reinforcement learning. No empirical results are provided.
Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy. Despite its success, the multi-step extensions of soft Q-learning remain relatively unexplored and limited to on-policy action sampling under the Boltzmann policy. In this brief research note, we first present a formal $n$-step formulation for soft Q-learning and then extend this framework to the fully off-policy case by introducing a novel Soft Tree Backup operator. Finally, we unify these developments into Soft $Q(λ)$, an elegant online, off-policy, eligibility trace framework that allows for efficient credit assignment under arbitrary behaviour policies. Our derivations propose a model-free method for learning entropy-regularised value functions that can be utilised in future empirical experiments.