Towards Optimal Pricing of Demand Response -- A Nonparametric Constrained Policy Optimization Approach
This addresses the need for stable and optimal pricing policies in safety-critical power systems, though it appears incremental as it builds on existing RL methods by removing restrictive assumptions.
The paper tackles the problem of optimizing electricity pricing for demand response to shift load from peak to off-peak hours, proposing a nonparametric constrained policy optimization approach that improves optimality and ensures stability, with experiments showing superior performance compared to state-of-the-art RL algorithms.
Demand response (DR) has been demonstrated to be an effective method for reducing peak load and mitigating uncertainties on both the supply and demand sides of the electricity market. One critical question for DR research is how to appropriately adjust electricity prices in order to shift electrical load from peak to off-peak hours. In recent years, reinforcement learning (RL) has been used to address the price-based DR problem because it is a model-free technique that does not necessitate the identification of models for end-use customers. However, the majority of RL methods cannot guarantee the stability and optimality of the learned pricing policy, which is undesirable in safety-critical power systems and may result in high customer bills. In this paper, we propose an innovative nonparametric constrained policy optimization approach that improves optimality while ensuring stability of the policy update, by removing the restrictive assumption on policy representation that the majority of the RL literature adopts: the policy must be parameterized or fall into a certain distribution class. We derive a closed-form expression of optimal policy update for each iteration and develop an efficient on-policy actor-critic algorithm to address the proposed constrained policy optimization problem. The experiments on two DR cases show the superior performance of our proposed nonparametric constrained policy optimization method compared with state-of-the-art RL algorithms.