Learning Memory Mechanisms for Decision Making through Demonstrations
This work addresses the problem of capturing memory processes in imitation learning for decision-making agents, offering a novel approach that is incremental in enhancing existing Transformer-based methods.
The paper tackles the challenge of integrating an agent's history into memory for decision-making in Partially Observable Markov Decision Processes by introducing memory dependency pairs to capture expert memory mechanisms, resulting in significant improvements over standard Transformers on tasks like Memory Gym and the Long-term Memory Benchmark.
In Partially Observable Markov Decision Processes, integrating an agent's history into memory poses a significant challenge for decision-making. Traditional imitation learning, relying on observation-action pairs for expert demonstrations, fails to capture the expert's memory mechanisms used in decision-making. To capture memory processes as demonstrations, we introduce the concept of memory dependency pairs $(p, q)$ indicating that events at time $p$ are recalled for decision-making at time $q$. We introduce AttentionTuner to leverage memory dependency pairs in Transformers and find significant improvements across several tasks compared to standard Transformers when evaluated on Memory Gym and the Long-term Memory Benchmark. Code is available at https://github.com/WilliamYue37/AttentionTuner.