PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?
For researchers studying LLM agents in interactive environments, this benchmark provides a new testbed for evaluating decision-making and self-evolution, though the findings are incremental.
The paper introduces PTCG-Bench, a benchmark for evaluating LLM agents in the Pokémon Trading Card Game, finding that while agents achieve non-trivial gameplay, sustained self-evolution remains challenging and performance is sensitive to harness design.
Given a strategically complex board game, human players can quickly learn to devise strategies after playing a few rounds. Autonomous agents require similar capabilities in realistic interactive environments, yet existing agent benchmarks often fail to fully capture such strategic and evolving decision-making scenarios. We present PTCG-Bench, a benchmark built on the Pok'{e}mon Trading Card Game (PTCG) that evaluates LLM agents at two complementary levels: (1) their decision-making performance within a single complex environment, and (2) their ability to self-evolving through accumulated experience. We further include a modular harness ablation to better interpret agent performance without conflating it with model capability. Our experiments show that, although LLM agents can achieve non-trivial gameplay performance, sustained and stable self-evolution remains challenging, and performance is sensitive to harness design. We hope that PTCG-Bench will facilitate future research on harness-aware and self-evolving agents in realistic interactive environments.