A storage expansion planning framework using reinforcement learning and simulation-based optimization
This addresses strategic planning for energy storage in microgrids, which is increasingly relevant for distributed generation, though it represents an incremental extension of existing reinforcement learning methods to a new application area.
The paper tackles the problem of long-term energy storage expansion planning for microgrids by developing a reinforcement learning and simulation-based optimization framework to determine optimal storage technology selection, timing, and capacity. The results show that the approach can derive better engineering solutions and identify optimal storage capacity thresholds that depend on price movements of storage units.
In the wake of the highly electrified future ahead of us, the role of energy storage is crucial wherever distributed generation is abundant, such as in microgrid settings. Given the variety of storage options that are becoming more and more economical, determining which type of storage technology to invest in, along with the appropriate timing and capacity becomes a critical research question. It is inevitable that these problems will continue to become increasingly relevant in the future and require strategic planning and holistic and modern frameworks in order to be solved. Reinforcement Learning algorithms have already proven to be successful in problems where sequential decision-making is inherent. In the operations planning area, these algorithms are already used but mostly in short-term problems with well-defined constraints. On the contrary, we expand and tailor these techniques to long-term planning by utilizing model-free algorithms combined with simulation-based models. A model and expansion plan have been developed to optimally determine microgrid designs as they evolve to dynamically react to changing conditions and to exploit energy storage capabilities. We show that it is possible to derive better engineering solutions that would point to the types of energy storage units which could be at the core of future microgrid applications. Another key finding is that the optimal storage capacity threshold for a system depends heavily on the price movements of the available storage units. By utilizing the proposed approaches, it is possible to model inherent problem uncertainties and optimize the whole streamline of sequential investment decision-making.