Boosting deep Reinforcement Learning using pretraining with Logical Options
This addresses misalignment in reinforcement learning agents for applications requiring long-term planning, though it is incremental as it builds on existing hybrid methods.
The paper tackles the problem of deep reinforcement learning agents being misaligned due to over-exploiting early rewards by proposing a hybrid approach that uses logical option-based pretraining to steer policies toward goal-directed behavior, resulting in improved long-horizon decision-making and outperforming various baselines.
Deep reinforcement learning agents are often misaligned, as they over-exploit early reward signals. Recently, several symbolic approaches have addressed these challenges by encoding sparse objectives along with aligned plans. However, purely symbolic architectures are complex to scale and difficult to apply to continuous settings. Hence, we propose a hybrid approach, inspired by humans' ability to acquire new skills. We use a two-stage framework that injects symbolic structure into neural-based reinforcement learning agents without sacrificing the expressivity of deep policies. Our method, called Hybrid Hierarchical RL (H^2RL), introduces a logical option-based pretraining strategy to steer the learning policy away from short-term reward loops and toward goal-directed behavior while allowing the final policy to be refined via standard environment interaction. Empirically, we show that this approach consistently improves long-horizon decision-making and yields agents that outperform strong neural, symbolic, and neuro-symbolic baselines.