RLAF: Reinforcement Learning from Automaton Feedback
This work addresses the challenge of reward engineering in reinforcement learning for tasks with temporal dependencies, providing a more automated solution for researchers and practitioners in AI.
The paper tackled the problem of reinforcement learning in environments with complex, history-dependent reward structures by introducing an approach that uses automaton-based feedback to generate preferences over trajectories, eliminating manual reward engineering. The results showed that this method outperformed traditional baselines in discrete and continuous environments, offering a scalable and efficient alternative with a convergence guarantee for near-optimal policies.
Reinforcement Learning (RL) in environments with complex, history-dependent reward structures poses significant challenges for traditional methods. In this work, we introduce a novel approach that leverages automaton-based feedback to guide the learning process, replacing explicit reward functions with preferences derived from a deterministic finite automaton (DFA). Unlike conventional approaches that use automata for direct reward specification, our method employs the structure of the DFA to generate preferences over trajectories that are used to learn a reward function, eliminating the need for manual reward engineering. Our framework introduces a static approach that uses the learned reward function directly for policy optimization and a dynamic approach that involves continuous refining of the reward function and policy through iterative updates until convergence. Our experiments in both discrete and continuous environments demonstrate that our approach enables the RL agent to learn effective policies for tasks with temporal dependencies, outperforming traditional reward engineering and automaton-based baselines such as reward machines and LTL-guided methods. Our results highlight the advantages of automaton-based preferences in handling non-Markovian rewards, offering a scalable, efficient, and human-independent alternative to traditional reward modeling. We also provide a convergence guarantee showing that under standard assumptions our automaton-guided preference-based framework learns a policy that is near-optimal with respect to the true non-Markovian objective.