Symmetry-Aware Transformer Training for Automated Planning
This addresses a key limitation for researchers and practitioners in automated planning by enabling more robust transformer-based methods, though it is incremental as it builds on prior work like PlanGPT.
The paper tackled the problem of transformers struggling with extrapolation in automated planning due to combinatorial symmetries in variable representations, and proposed a symmetry-aware contrastive learning objective that effectively and efficiently improved performance over PlanGPT across multiple domains.
While transformers excel in many settings, their application in the field of automated planning is limited. Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to hard planning problems. This in turn stems from problem symmetries: planning tasks can be represented with arbitrary variable names that carry no meaning beyond being identifiers. This causes a combinatorial explosion of equivalent representations that pure transformers cannot efficiently learn from. We propose a novel contrastive learning objective to make transformers symmetry-aware and thereby compensate for their lack of inductive bias. Combining this with architectural improvements, we show that transformers can be efficiently trained for either plan-generation or heuristic-prediction. Our results across multiple planning domains demonstrate that our symmetry-aware training effectively and efficiently addresses the limitations of PlanGPT.