Collect & Infer -- a fresh look at data-efficient Reinforcement Learning
It addresses data-efficiency for reinforcement learning researchers, but is a position paper proposing a conceptual framework rather than empirical results, making it incremental.
This paper tackles the problem of data-efficiency in reinforcement learning by proposing a 'Collect and Infer' paradigm that emphasizes both data collection and knowledge inference, arguing that current methods understudy collection aspects.
This position paper proposes a fresh look at Reinforcement Learning (RL) from the perspective of data-efficiency. Data-efficient RL has gone through three major stages: pure on-line RL where every data-point is considered only once, RL with a replay buffer where additional learning is done on a portion of the experience, and finally transition memory based RL, where, conceptually, all transitions are stored and re-used in every update step. While inferring knowledge from all explicitly stored experience has lead to a tremendous gain in data-efficiency, the question of how this data is collected has been vastly understudied. We argue that data-efficiency can only be achieved through careful consideration of both aspects. We propose to make this insight explicit via a paradigm that we call 'Collect and Infer', which explicitly models RL as two separate but interconnected processes, concerned with data collection and knowledge inference respectively. We discuss implications of the paradigm, how its ideas are reflected in the literature, and how it can guide future research into data efficient RL.