Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration
This addresses convergence issues in offline reinforcement learning for practitioners, though it is incremental as it builds on existing entropy-regularized methods.
The paper tackled the instability of soft fitted Q-iteration under function approximation and distribution shift by showing that the soft Bellman operator contracts in a stationary norm, and introduced stationary-reweighted soft FQI to restore local linear convergence with geometrically damped errors.
Fitted Q-iteration (FQI) and its entropy-regularized variant, soft FQI, are central tools for value-based model-free offline reinforcement learning, but can behave poorly under function approximation and distribution shift. In the entropy-regularized setting, we show that the soft Bellman operator is locally contractive in the stationary norm of the soft-optimal policy, rather than in the behavior norm used by standard FQI. This geometric mismatch explains the instability of soft Q-iteration with function approximation in the absence of Bellman completeness. To restore contraction, we introduce stationary-reweighted soft FQI, which reweights each regression update using the stationary distribution of the current policy. We prove local linear convergence under function approximation with geometrically damped weight-estimation errors, assuming approximate realizability. Our analysis further suggests that global convergence may be recovered by gradually reducing the softmax temperature, and that this continuation approach can extend to the hardmax limit under a mild margin condition.