Actor Loss of Soft Actor Critic Explained
This is an incremental technical clarification for researchers implementing SAC, addressing a specific mathematical detail in reinforcement learning.
The paper explains the derivation of the actor loss function in Soft Actor-Critic (SAC), including the mathematical background and gradient estimation, while comparing the reparameterization trick with the nabla log trick and raising questions about efficiency.
This technical report is devoted to explaining how the actor loss of soft actor critic is obtained, as well as the associated gradient estimate. It gives the necessary mathematical background to derive all the presented equations, from the theoretical actor loss to the one implemented in practice. This necessitates a comparison of the reparameterization trick used in soft actor critic with the nabla log trick, which leads to open questions regarding the most efficient method to use.