Graph Constrained Reinforcement Learning for Natural Language Action Spaces
This addresses the problem of scalable exploration in natural language action spaces for AI agents in interactive simulations, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of reinforcement learning in text-based Interactive Fiction games with large natural language action spaces by introducing KG-A2C, an agent that uses a dynamic knowledge graph for state reasoning and action generation, resulting in outperformance over current agents despite exponential action space growth.
Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.