Auxiliary Reward Generation with Transition Distance Representation Learning
This work addresses the challenge of automating reward design in RL for real-world problems, offering a solution that reduces human effort and bias, though it appears incremental as it builds on existing representation learning techniques.
The paper tackles the problem of labor-intensive and biased human-designed reward functions in reinforcement learning by proposing a representation learning approach to measure transition distance between states and generate auxiliary rewards automatically. The method improves learning efficiency and convergent stability in manipulation tasks, as demonstrated by experimental results.
Reinforcement learning (RL) has shown its strength in challenging sequential decision-making problems. The reward function in RL is crucial to the learning performance, as it serves as a measure of the task completion degree. In real-world problems, the rewards are predominantly human-designed, which requires laborious tuning, and is easily affected by human cognitive biases. To achieve automatic auxiliary reward generation, we propose a novel representation learning approach that can measure the ``transition distance'' between states. Building upon these representations, we introduce an auxiliary reward generation technique for both single-task and skill-chaining scenarios without the need for human knowledge. The proposed approach is evaluated in a wide range of manipulation tasks. The experiment results demonstrate the effectiveness of measuring the transition distance between states and the induced improvement by auxiliary rewards, which not only promotes better learning efficiency but also increases convergent stability.