Adjusting the Output of Decision Transformer with Action Gradient
This work addresses stability issues in offline RL for DT-based algorithms, offering an incremental improvement over existing methods.
The paper tackled the challenges of trajectory stitching and action extrapolation in Decision Transformer (DT) for offline reinforcement learning by proposing Action Gradient (AG), which directly adjusts actions using Q-value gradients, resulting in significant performance enhancements and some state-of-the-art results.
Decision Transformer (DT), which integrates reinforcement learning (RL) with the transformer model, introduces a novel approach to offline RL. Unlike classical algorithms that take maximizing cumulative discounted rewards as objective, DT instead maximizes the likelihood of actions. This paradigm shift, however, presents two key challenges: stitching trajectories and extrapolation of action. Existing methods, such as substituting specific tokens with predictive values and integrating the Policy Gradient (PG) method, address these challenges individually but fail to improve performance stably when combined due to inherent instability. To address this, we propose Action Gradient (AG), an innovative methodology that directly adjusts actions to fulfill a function analogous to that of PG, while also facilitating efficient integration with token prediction techniques. AG utilizes the gradient of the Q-value with respect to the action to optimize the action. The empirical results demonstrate that our method can significantly enhance the performance of DT-based algorithms, with some results achieving state-of-the-art levels.