Automatic formation of the structure of abstract machines in hierarchical reinforcement learning with state clustering
This work addresses the challenge of task decomposition for real robots in hierarchical reinforcement learning, though it appears incremental as it builds upon the existing HAM framework.
The paper tackles the problem of automatic hierarchy formation in hierarchical reinforcement learning by introducing an internal environment where state represents the hierarchy structure, using Q-learning to optimize it, and extending the HAM framework with an on-model approach for sub-machine selection. Preliminary experiments showed promising results.
We introduce a new approach to hierarchy formation and task decomposition in hierarchical reinforcement learning. Our method is based on the Hierarchy Of Abstract Machines (HAM) framework because HAM approach is able to design efficient controllers that will realize specific behaviors in real robots. The key to our algorithm is the introduction of the internal or "mental" environment in which the state represents the structure of the HAM hierarchy. The internal action in this environment leads to changes the hierarchy of HAMs. We propose the classical Q-learning procedure in the internal environment which allows the agent to obtain an optimal hierarchy. We extends the HAM framework by adding on-model approach to select the appropriate sub-machine to execute action sequences for certain class of external environment states. Preliminary experiments demonstrated the prospects of the method.