Automaton Distillation: Neuro-Symbolic Transfer Learning for Deep Reinforcement Learning
This addresses sample efficiency and generalization issues in reinforcement learning for sequential decision tasks, representing an incremental improvement.
The paper tackles the high sample cost and poor generalization of deep reinforcement learning by introducing automaton distillation, a neuro-symbolic transfer learning method that distills Q-value estimates from a teacher into an automaton to bootstrap learning in new environments, resulting in decreased time to find optimal policies.
Reinforcement learning (RL) is a powerful tool for finding optimal policies in sequential decision processes. However, deep RL methods have two weaknesses: collecting the amount of agent experience required for practical RL problems is prohibitively expensive, and the learned policies exhibit poor generalization on tasks outside the training data distribution. To mitigate these issues, we introduce automaton distillation, a form of neuro-symbolic transfer learning in which Q-value estimates from a teacher are distilled into a low-dimensional representation in the form of an automaton. We then propose methods for generating Q-value estimates where symbolic information is extracted from a teacher's Deep Q-Network (DQN). The resulting Q-value estimates are used to bootstrap learning in the target discrete and continuous environment via a modified DQN and Twin-Delayed Deep Deterministic (TD3) loss function, respectively. We demonstrate that automaton distillation decreases the time required to find optimal policies for various decision tasks in new environments, even in a target environment different in structure from the source environment.