Siqi Bu

Ziqing Zhu, Ze Hu, Ka Wing Chan et al.

The increasing penetration of renewable generations, along with the deregulation and marketization of power industry, promotes the transformation of power market operation paradigms. The optimal bidding strategy and dispatching methodology under these new paradigms are prioritized concerns for both market participants and power system operators, with obstacles of uncertain characteristics, computational efficiency, as well as requirements of hyperopic decision-making. To tackle these problems, the Reinforcement Learning (RL), as an emerging machine learning technique with advantages compared with conventional optimization tools, is playing an increasingly significant role in both academia and industry. This paper presents a comprehensive review of RL applications in deregulated power market operation including bidding and dispatching strategy optimization, based on more than 150 carefully selected literatures. For each application, apart from a paradigmatic summary of generalized methodology, in-depth discussions of applicability and obstacles while deploying RL techniques are also provided. Finally, some RL techniques that have great potentiality to be deployed in bidding and dispatching problems are recommended and discussed.

Qiang Fu, Siqi Bu, Zijun Bin et al.

It has been well known that inverter-based resources (IBRs) can lead to converter-driven stability issues under weak grid connections. However, as the number of IBRs increases, instabilities can also occur even under strong grid connections. A practical case is presented to demonstrate this conclusion, using large-scale energy storage systems (ESSs) as an example. In this study, the ESSs induce oscillations with a frequency of 150 Hz in the d-q coordinates while providing both capacitive and inductive reactive power support (achieved by ESS functional control loops) to the connected power system. Theoretical analysis reveals that under strong grid connections, the dynamic interactions among power conversion systems (PCSs) of ESSs can be superimposed and intensified as the ESS scale extends, which reduces oscillation damping and leads to system instability. This indicates that ESS functional control loops also have potential instability risks when providing supports to power systems, which should be carefully examined. Finally, major impact factors are identified to mitigate the oscillations, and the conclusions are validated based on the SIMULINK platform. This paper provides valuable practical insights into system instabilities even under strong grid conditions, emphasizing the importance of functional control design and careful planning of the scale for IBR-dominated systems.

Qiang Fu, Wenjuan Du, Siqi Bu et al.

Massive controlled DC resources (CDCRs), such as battery energy storage systems, are connected to AC power systems through bidirectional inverters for power balance requirements. This study investigates converter-driven stability (CDS) issues in the sub-synchronous frequency range caused by large-scale bidirectional inverter-based stations (IBSs). The impacts of the AC and DC connections of IBSs on subsynchronous oscillations (SSOs) are compared by examining three factors: the number of CDCRs, power flow direction, and control parameters of the inverters. For AC connections, IBSs may induce instability as the number of CDCRs increases, regardless of the power flow direction. To maintain stability, the maximum power amplitude of the IBS is calculated. It is found that switching to DC connections can reduce these instability risks if the DC line resistance is much less than the AC line reactance. Moreover, the method of tuning control parameters is demonstrated to be more effective in improving power-related critical stability under DC connections. Therefore, The DC-IBS is preferred for high-voltage transmission. Finally, the conclusions are validated in power systems connected with both AC- and DC-IBSs under various network topologies and system scales.

Qiang Fu, Siqi Bu, Yang Wang et al.

This study investigates the small-signal stability of centralized energy storage systems (CESSs) using grid-following (GFL) and grid-forming (GFM) controls, particularly focusing on bidirectional power flow and multiple energy storage systems (ESSs). To address the issue of complex dynamics in CESSs when comprehensive GFL and GFM control loops are considered, high-order dynamics are simplified using the virtual damping method by focusing on the dominant oscillation mode. Damping analysis verifies that CESSs using a single-type control (either GFL or GFM) have dynamic superimposition characteristics. Specifically, as ESS number increases, the damping of GFM-CESSs improves but that of GFL-CESSs decreases. The damping sensitivity shows that the damping of GFM-CESSs is more sensitive to bidirectional power flow and all control loops, whereas that of GFL-CESSs is more sensitive to d-axis control loop. Consequently, GFM-CESSs are preferred for large-scale integration but are limited in scenarios with significant power reversal. If GFL and GFM controls are hybridized in CESSs, the ratio of GFM-CESSs should be constrained to avoid instability from modal resonance between GFL-CESSs and GFM-CESSs. This highlights that implementing GFM-CESSs necessitates considering scenario limitations rather than pursuing maximal integration under hybrid integration conditions. The conclusions are validated through modal analysis and time-domain simulations.

Ziqing Zhu, Ka Wing Chan, Siqi Bu et al.

The Nash Equilibrium (NE) estimation in bidding games of electricity markets is the key concern of both generation companies (GENCOs) for bidding strategy optimization and the Independent System Operator (ISO) for market surveillance. However, existing methods for NE estimation in emerging modern electricity markets (FEM) are inaccurate and inefficient because the priori knowledge of bidding strategies before any environment changes, such as load demand variations, network congestion, and modifications of market design, is not fully utilized. In this paper, a Bayes-adaptive Markov Decision Process in FEM (BAMDP-FEM) is therefore developed to model the GENCOs' bidding strategy optimization considering the priori knowledge. A novel Multi-Agent Generative Adversarial Imitation Learning algorithm (MAGAIL-FEM) is then proposed to enable GENCOs to learn simultaneously from priori knowledge and interactions with changing environments. The obtained NE is a Bayesian Nash Equilibrium (BNE) with priori knowledge transferred from the previous environment. In the case study, the superiority of this proposed algorithm in terms of convergence speed compared with conventional methods is verified. It is concluded that the optimal bidding strategies in the obtained BNE can always lead to more profits than NE due to the effective learning from the priori knowledge. Also, BNE is more accurate and consistent with situations in real-world markets.

Ziqing Zhu, Siqi Bu, Ka Wing Chan et al.

This two-part paper develops a paradigmatic theory and detailed methods of the joint electricity market design using reinforcement-learning (RL)-based simulation. In Part 2, this theory is further demonstrated by elaborating detailed methods of designing an electricity spot market (ESM), together with a reserved capacity product (RC) in the ancillary service market (ASM) and a virtual bidding (VB) product in the financial market (FM). Following the theory proposed in Part 1, firstly, market design options in the joint market are specified. Then, the Markov game model is developed, in which we show how to incorporate market design options and uncertain risks in model formulation. A multi-agent policy proximal optimization (MAPPO) algorithm is elaborated, as a practical implementation of the generalized market simulation method developed in Part 1. Finally, the case study demonstrates how to pick the best market design options by using some of the market operation performance indicators proposed in Part 1, based on the simulation results generated by implementing the MAPPO algorithm. The impacts of different market design options on market participants' bidding strategy preference are also discussed.

Ziqing Zhu, Siqi Bu, Ka Wing Chan et al.

In face of the pressing need of decarbonization in the power sector, the re-design of electricity market is necessary as a Marco-level approach to accommodate the high penetration of renewable generations, and to achieve power system operation security, economic efficiency, and environmental friendliness. However, existing market design methodologies suffer from the lack of coordination among energy spot market (ESM), ancillary service market (ASM) and financial market (FM), i.e., the "joint market", and the lack of reliable simulation-based verification. To tackle these deficiencies, this two-part paper develops a paradigmatic theory and detailed methods of the joint market design using reinforcement-learning (RL)-based simulation. In Part 1, the theory and framework of this novel market design philosophy are proposed. First, the controversial market design options while designing the joint market are summarized as the targeted research questions. Second, the Markov game model is developed to describe the bidding game in the joint market, incorporating the market design options to be determined. Third, a framework of deploying multiple types of RL algorithms to simulate the market model is developed. Finally, several market operation performance indicators are proposed to validate the market design based on the simulation results.