Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization
This addresses the issue of unnatural behavior in RL agents for improved interpretability and trustworthiness, though it appears incremental as it builds on existing RL methods with a novel framework.
The paper tackles the problem of creating human-like reinforcement learning agents by formulating human-likeness as trajectory optimization and introducing Macro Action Quantization (MAQ), which distills human demonstrations into macro actions; experiments on D4RL Adroit benchmarks show MAQ significantly improves human-likeness, achieving the highest rankings in human evaluation.
Human-like agents have long been one of the goals in pursuing artificial intelligence. Although reinforcement learning (RL) has achieved superhuman performance in many domains, relatively little attention has been focused on designing human-like RL agents. As a result, many reward-driven RL agents often exhibit unnatural behaviors compared to humans, raising concerns for both interpretability and trustworthiness. To achieve human-like behavior in RL, this paper first formulates human-likeness as trajectory optimization, where the objective is to find an action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding-horizon control to human-like learning as a tractable and efficient implementation. To achieve this, we introduce Macro Action Quantization (MAQ), a human-like RL framework that distills human demonstrations into macro actions via Vector-Quantized VAE. Experiments on D4RL Adroit benchmarks show that MAQ significantly improves human-likeness, increasing trajectory similarity scores, and achieving the highest human-likeness rankings among all RL agents in the human evaluation study. Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, opening a promising direction for learning human-like RL agents. Our code is available at https://rlg.iis.sinica.edu.tw/papers/MAQ.