Spectral Analysis of Heavy-Ball Q-value Iteration
Theoretical analysis of momentum acceleration in Q-learning, addressing a gap in understanding for control tasks.
This paper provides a convergence analysis of heavy-ball Q-value iteration using switched linear system theory and joint spectral radius, identifying conditions under which momentum accelerates Q-learning.
We study the convergence and acceleration of Q-value iteration (QVI) with momentum, or heavy-ball QVI. Although acceleration of value iteration has been studied extensively, there has been less work on the acceleration of heavy-ball QVI for control tasks. We analyze heavy-ball QVI from the viewpoint of switched linear system (SLS) theory and the joint spectral radius (JSR). First, we convert heavy-ball QVI into an exact SLS and interpret its convergence through the JSR. We also study, in JSR-based terms, conditions under which heavy-ball QVI can be faster than standard QVI.