Graph Decision Transformer
This addresses the problem of learning policies from static data in offline RL, offering a novel approach that could benefit AI applications in robotics and gaming, though it appears incremental as it builds on existing transformer-based methods.
The paper tackles the challenge of offline reinforcement learning by proposing Graph Decision Transformer (GDT), which models input sequences as causal graphs to capture dependencies and improve learning; experiments show it matches or surpasses state-of-the-art methods on Atari and OpenAI Gym benchmarks.
Offline reinforcement learning (RL) is a challenging task, whose objective is to learn policies from static trajectory data without interacting with the environment. Recently, offline RL has been viewed as a sequence modeling problem, where an agent generates a sequence of subsequent actions based on a set of static transition experiences. However, existing approaches that use transformers to attend to all tokens naively can overlook the dependencies between different tokens and limit long-term dependency learning. In this paper, we propose the Graph Decision Transformer (GDT), a novel offline RL approach that models the input sequence into a causal graph to capture potential dependencies between fundamentally different concepts and facilitate temporal and causal relationship learning. GDT uses a graph transformer to process the graph inputs with relation-enhanced mechanisms, and an optional sequence transformer to handle fine-grained spatial information in visual tasks. Our experiments show that GDT matches or surpasses the performance of state-of-the-art offline RL methods on image-based Atari and OpenAI Gym.