Visualizing Latent Phase Structures in Locomotion Policies: A Multi-Environment Study with Temporal Feature Extension
This work addresses the challenge of interpreting learned locomotion policies in DRL, offering a visualization tool for researchers studying motion control, though the improvement is incremental over existing clustering approaches.
The authors propose a framework to visualize latent motion phase structures in locomotion policies by extending clustering features to include actions, next states, and next actions, and introducing a cluster count determination method that suppresses self-transitions. Applied to MuJoCo benchmarks (Ant-v5, HalfCheetah-v5, Walker2D-v5), the method identifies phase structures with clearer and more regular transition rules than existing methods.
Deep reinforcement learning (DRL) has been shown to achieve high performance on locomotion control tasks in MuJoCo benchmarks such as HalfCheetah, Ant, and Walker2D. However, visualizing the motion structures internally obtained by a trained policy function implemented as a deep neural network remains challenging. It is known from biomechanics and related fields that locomotion control is realized through the repetition of motion phases such as the stance phase and swing phase. In this study, we propose a framework for uncovering latent motion phase structures from trajectories generated by locomotion control policies through interaction with the environment. The proposed method extends the clustering features from state observations alone to augmented features including actions, next states, and next actions, and introduces a method for determining the number of clusters that suppresses self-transitions. Applying the proposed method to three environments -- Ant-v5, HalfCheetah-v5, and Walker2D-v5 -- we successfully identified phase structures with clearer and more regular transition rules than those obtained by the existing method.