Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
This addresses safety concerns for balance responsible parties using RL in energy markets, but it is incremental as it builds on existing RL methods with a post-processing step.
The paper tackles the safety issue of reinforcement learning policies in real-world energy arbitrage by proposing a control framework that optimizes revenue and then corrects the policy using knowledge distillation, achieving deployment on a real battery with evaluation on 2023 Belgian imbalance prices.
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.