Efficient entity-based reinforcement learning
This work addresses a bottleneck in DRL for real-world applications like robotics and logistics, where observations involve variable numbers of entities, offering a novel method to handle structured data, though it is incremental as it builds on existing techniques.
The paper tackles the problem of applying deep reinforcement learning to decision-making tasks with structured, entity-based observations, which are incompatible with standard architectures, by proposing a method that combines set representations, slot attention, and graph neural networks. The result is an efficient and scalable approach that significantly improves training time and robustness, demonstrated on environments like Atari Learning Environment and Simple Playgrounds.
Recent deep reinforcement learning (DRL) successes rely on end-to-end learning from fixed-size observational inputs (e.g. image, state-variables). However, many challenging and interesting problems in decision making involve observations or intermediary representations which are best described as a set of entities: either the image-based approach would miss small but important details in the observations (e.g. ojects on a radar, vehicles on satellite images, etc.), the number of sensed objects is not fixed (e.g. robotic manipulation), or the problem simply cannot be represented in a meaningful way as an image (e.g. power grid control, or logistics). This type of structured representations is not directly compatible with current DRL architectures, however, there has been an increase in machine learning techniques directly targeting structured information, potentially addressing this issue. We propose to combine recent advances in set representations with slot attention and graph neural networks to process structured data, broadening the range of applications of DRL algorithms. This approach allows to address entity-based problems in an efficient and scalable way. We show that it can improve training time and robustness significantly, and demonstrate their potential to handle structured as well as purely visual domains, on multiple environments from the Atari Learning Environment and Simple Playgrounds.