Sample-efficient Neuro-symbolic Proximal Policy Optimization
This work addresses sample inefficiency and sparse-reward challenges in deep reinforcement learning for tasks with long planning horizons and multiple sub-goals.
The authors propose a neuro-symbolic extension of PPO that transfers logical policy specifications from easier instances to guide learning in harder ones, achieving faster learning and higher returns on three benchmarks compared to PPO and a Reward Machine baseline.
Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.