Hyperspace Neighbor Penetration Approach to Dynamic Programming for Model-Based Reinforcement Learning Problems with Slowly Changing Variables in A Continuous State Space
This work addresses a specific bottleneck in reinforcement learning for applications like climate control systems, where variables change slowly, but it is incremental as it builds on existing dynamic programming methods with a novel adaptation.
The paper tackles the problem of handling slowly changing variables in continuous state spaces for model-based reinforcement learning, which classical methods like Dynamic Programming with Tile Coding fail to address due to computational inefficiency. It introduces the Hyperspace Neighbor Penetration (HNP) approach, which captures tiny changes by allowing state penetration into neighboring hyper-tiles, enabling the use of a coarse grid system and achieving orders of magnitude greater efficiency.
Slowly changing variables in a continuous state space constitute an important category of reinforcement learning and see its application in many domains, such as modeling a climate control system where temperature, humidity, etc. change slowly over time. However, this subject is less addressed in recent studies. Classical methods with certain variants, such as Dynamic Programming with Tile Coding which discretizes the state space, fail to handle slowly changing variables because those methods cannot capture the tiny changes in each transition step, as it is computationally expensive or impossible to establish an extremely granular grid system. In this paper, we introduce a Hyperspace Neighbor Penetration (HNP) approach that solves the problem. HNP captures in each transition step the state's partial "penetration" into its neighboring hyper-tiles in the gridded hyperspace, thus does not require the transition to be inter-tile in order for the change to be captured. Therefore, HNP allows for a very coarse grid system, which makes the computation feasible. HNP assumes near linearity of the transition function in a local space, which is commonly satisfied. In summary, HNP can be orders of magnitude more efficient than classical method in handling slowly changing variables in reinforcement learning. We have made an industrial implementation of NHP with a great success.