The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure
This work addresses scalability issues in RL for practitioners, but it is incremental as it builds on existing low-rank and transfer learning concepts.
The paper tackles the high computational cost of reinforcement learning (RL) in large state and action spaces by studying transfer RL with latent low-rank structure, showing that their algorithm reduces regret bounds by removing dependence on S, A, or SA and achieves minimax-optimality with respect to a transferability coefficient α in most settings.
Many reinforcement learning (RL) algorithms are too costly to use in practice due to the large sizes $S, A$ of the problem's state and action space. To resolve this issue, we study transfer RL with latent low rank structure. We consider the problem of transferring a latent low rank representation when the source and target MDPs have transition kernels with Tucker rank $(S , d, A )$, $(S , S , d), (d, S, A )$, or $(d , d , d )$. In each setting, we introduce the transfer-ability coefficient $α$ that measures the difficulty of representational transfer. Our algorithm learns latent representations in each source MDP and then exploits the linear structure to remove the dependence on $S, A $, or $S A$ in the target MDP regret bound. We complement our positive results with information theoretic lower bounds that show our algorithms (excluding the ($d, d, d$) setting) are minimax-optimal with respect to $α$.