A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learning
This is an incremental improvement for multi-task reinforcement learning systems aiming to reduce data acquisition needs.
The paper tackles the problem of data inefficiency in multi-task reinforcement learning by modeling Q-functions as a tensor and imposing a low-rank condition to infer task similarities, demonstrating efficiency in benchmark and practical experiments.
In pursuit of reinforcement learning systems that could train in physical environments, we investigate multi-task approaches as a means to alleviate the need for massive data acquisition. In a tabular scenario where the Q-functions are collected across tasks, we model our learning problem as optimizing a higher order tensor structure. Recognizing that close-related tasks may require similar actions, our proposed method imposes a low-rank condition on this aggregated Q-tensor. The rationale behind this approach to multi-task learning is that the low-rank structure enforces the notion of similarity, without the need to explicitly prescribe which tasks are similar, but inferring this information from a reduced amount of data simultaneously with the stochastic optimization of the Q-tensor. The efficiency of our low-rank tensor approach to multi-task learning is demonstrated in two numerical experiments, first in a benchmark environment formed by a collection of inverted pendulums, and then into a practical scenario involving multiple wireless communication devices.