Value Iteration with Options and State Aggregation
This work addresses efficiency improvements in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing frameworks like options and state aggregation.
The paper tackles the problem of solving Markov Decision Processes more efficiently by combining state aggregation (state abstraction) with the options framework (temporal abstraction), showing that their integration yields greater benefits than using either alone. It introduces a hierarchical value iteration algorithm that first coarsely solves subgoals and then uses these approximations to exactly solve the MDP, resulting in faster solutions on several problems compared to vanilla value iteration.
This paper presents a way of solving Markov Decision Processes that combines state abstraction and temporal abstraction. Specifically, we combine state aggregation with the options framework and demonstrate that they work well together and indeed it is only after one combines the two that the full benefit of each is realized. We introduce a hierarchical value iteration algorithm where we first coarsely solve subgoals and then use these approximate solutions to exactly solve the MDP. This algorithm solved several problems faster than vanilla value iteration.