2.3CRJul 26, 2024Code
HADES: Detecting Active Directory Attacks via Whole Network Provenance AnalyticsQi Liu, Kaibin Bao, Wajih Ul Hassan et al.
Due to its crucial role in identity and access management in modern enterprise networks, Active Directory (AD) is a top target of Advanced Persistence Threat (APT) actors. Conventional intrusion detection systems (IDS) excel at identifying malicious behaviors caused by malware, but often fail to detect stealthy attacks launched by APT actors. Recent advance in provenance-based IDS (PIDS) shows promises by exposing malicious system activities in causal attack graphs. However, existing approaches are restricted to intra-machine tracing, and unable to reveal the scope of attackers' traversal inside a network. We propose HADES, the first PIDS capable of performing accurate causality-based cross-machine tracing by leveraging a novel concept called logon session based execution partitioning to overcome several challenges in cross-machine tracing. We design HADES as an efficient on-demand tracing system, which performs whole-network tracing only when it first identifies an authentication anomaly signifying an ongoing AD attack, for which we introduce a novel lightweight authentication anomaly detection model rooted in our extensive analysis of AD attacks. To triage attack alerts, we present a new algorithm integrating two key insights we identified in AD attacks. Our evaluations show that HADES outperforms both popular open source detection systems and a prominent commercial AD attack detector.
CyberCertBench: Evaluating LLMs in Cybersecurity Certification KnowledgeGustav Keppler, Ghada Elbez, Veit Hagenmeyer
The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against industry standards. We introduceCyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks derived from industry recognized certifications. CyberCertBench evaluates LLM domain knowledgeagainst the professional standards of Information Technology cybersecurity and more specializedareas such as Operational Technology and related cybersecurity standards. Concurrently, we propose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable,natural language explanations for model performance. Our evaluation shows that frontier modelsachieve human expert level in general networking and IT security knowledge. However, theiraccuracy declines in questions that require vendor-specific nuances or knowledge in formalstandards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstratesremarkable gains in parameter efficiency, while recent larger models show diminishing returns.Code and evaluation scripts are available at: https://github.com/GKeppler/CyberCertBench.
2.4OCAug 23, 2018
A Generalized Framework for Chance-constrained Optimal Power FlowTillmann Mühlpfordt, Timm Faulwasser, Veit Hagenmeyer
Deregulated energy markets, demand forecasting, and the continuously increasing share of renewable energy sources call---among others---for a structured consideration of uncertainties in optimal power flow problems. The main challenge is to guarantee power balance while maintaining economic and secure operation. In the presence of Gaussian uncertainties affine feedback policies are known to be viable options for this task. The present paper advocates a general framework for chance-constrained OPF problems in terms of continuous random variables. It is shown that, irrespective of the type of distribution, the random-variable minimizers lead to affine feedback policies. Introducing a three-step methodology that exploits polynomial chaos expansion, the present paper provides a constructive approach to chance-constrained optimal power flow problems that does not assume a specific distribution, e.g. Gaussian, for the uncertainties. We illustrate our findings by means of a tutorial example and a 300-bus test case.
Transformer Training Strategies for Forecasting Multiple Load Time SeriesMatthias Hertel, Maximilian Beichter, Benedikt Heidrich et al.
In the smart grid of the future, accurate load forecasts on the level of individual clients can help to balance supply and demand locally and to prevent grid outages. While the number of monitored clients will increase with the ongoing smart meter rollout, the amount of data per client will always be limited. We evaluate whether a Transformer load forecasting model benefits from a transfer learning strategy, where a global univariate model is trained on the load time series from multiple clients. In experiments with two datasets containing load time series from several hundred clients, we find that the global training strategy is superior to the multivariate and local training strategies used in related work. On average, the global training strategy results in 21.8% and 12.8% lower forecasting errors than the two other strategies, measured across forecasting horizons from one day to one month into the future. A comparison to linear models, multi-layer perceptrons and LSTMs shows that Transformers are effective for load forecasting when they are trained with the global training strategy.
1.2SYJan 23, 2017
Nearest-Neighbor Based Non-Parametric Probabilistic Forecasting with Applications in Photovoltaic SystemsJorge Ángel González Ordiano, Wolfgang Doneit, Simon Waczowicz et al.
The present contribution offers a simple methodology for the obtainment of data-driven interval forecasting models by combining pairs of quantile regressions. Those regressions are created without the usage of the non-differentiable pinball-loss function, but through a k-nearest-neighbors based training set transformation and traditional regression approaches. By leaving the underlying training algorithms of the data mining techniques unchanged, the presented approach simplifies the creation of quantile regressions with more complex techniques (e.g. artificial neural networks). The quality of the presented methodology is tested on the usecase of photovoltaic power forecasting, for which quantile regressions using polynomial models as well as artificial neural networks and support vector regressions are created. From the resulting evaluation values it can be concluded that acceptable interval forecasting models are created.
1.8LGDec 13, 2022
AutoPV: Automated photovoltaic forecasts with limited information using an ensemble of pre-trained modelsStefan Meisenbacher, Benedikt Heidrich, Tim Martin et al.
Accurate PhotoVoltaic (PV) power generation forecasting is vital for the efficient operation of Smart Grids. The automated design of such accurate forecasting models for individual PV plants includes two challenges: First, information about the PV mounting configuration (i.e. inclination and azimuth angles) is often missing. Second, for new PV plants, the amount of historical data available to train a forecasting model is limited (cold-start problem). We address these two challenges by proposing a new method for day-ahead PV power generation forecasts called AutoPV. AutoPV is a weighted ensemble of forecasting models that represent different PV mounting configurations. This representation is achieved by pre-training each forecasting model on a separate PV plant and by scaling the model's output with the peak power rating of the corresponding PV plant. To tackle the cold-start problem, we initially weight each forecasting model in the ensemble equally. To tackle the problem of missing information about the PV mounting configuration, we use new data that become available during operation to adapt the ensemble weights to minimize the forecasting error. AutoPV is advantageous as the unknown PV mounting configuration is implicitly reflected in the ensemble weights, and only the PV plant's peak power rating is required to re-scale the ensemble's output. AutoPV also allows to represent PV plants with panels distributed on different roofs with varying alignments, as these mounting configurations can be reflected proportionally in the weighting. Additionally, the required computing memory is decoupled when scaling AutoPV to hundreds of PV plants, which is beneficial in Smart Grids with limited computing capabilities. For a real-world data set with 11 PV plants, the accuracy of AutoPV is comparable to a model trained on two years of data and outperforms an incrementally trained model.
1.2SYMay 24, 2018
Storage Scheduling with Stochastic Uncertainties: Feasibility and Cost of ImbalancesR. R. Appino, J. Á. González Ordiano, R. Mikut et al.
Dispatchability of renewable energy sources and inflexible loads can be achieved using a volatility-compensating energy storage. However, as the future power outputs of the inflexible devices are uncertain, the computation of a dispatch schedule for such aggregated systems is non-trivial. In the present paper, we propose a novel scheduling method that enforces the feasibility of the dispatch schedule with a pre-determined probability based on a description of the operation of the system as a two-stage decision process. Thereby, a crucial point is the use of probabilistic forecasts, in terms of cumulative density function, of the inflexible energy consumption/production profile. Then, for the sake of comparison, we introduce a second scheduling method based on state-of-the-art scenario optimization, where, unlike the proposed method, the focus is on the minimization of the expected final cost. We draw upon simulations based on real consumption and production data to compare the methods and illustrate our findings.
1.2APSep 27, 2022
Predicting the power grid frequency of European islandsThorbjørn Lund Onsaker, Heidi S. Nygård, Damià Gomila et al.
Modelling, forecasting and overall understanding of the dynamics of the power grid and its frequency are essential for the safe operation of existing and future power grids. Much previous research was focused on large continental areas, while small systems, such as islands are less well-studied. These natural island systems are ideal testing environments for microgrid proposals and artificially islanded grid operation. In the present paper, we utilize measurements of the power grid frequency obtained in European islands: the Faroe Islands, Ireland, the Balearic Islands and Iceland and investigate how their frequency can be predicted, compared to the Nordic power system, acting as a reference. The Balearic islands are found to be particularly deterministic and easy to predict in contrast to hard-to-predict Iceland. Furthermore, we show that typically 2-4 weeks of data are needed to improve prediction performance beyond simple benchmarks.
2.4OCJun 14, 2018
The Price of Uncertainty: Chance-constrained OPF vs. In-hindsight OPFTillmann Mühlpfordt, Veit Hagenmeyer, Timm Faulwasser
The operation of power systems has become more challenging due to feed-in of volatile renewable energy sources. Chance-constrained optimal power flow (ccOPF) is one possibility to explicitly consider volatility via probabilistic uncertainties resulting in mean-optimal feedback policies. These policies are computed before knowledge of the realization of the uncertainty is available. On the other hand, the hypothetical case of computing the power injections knowing every realization beforehand---called in-hindsight OPF(hOPF)---cannot be outperformed w.r.t. costs and constraint satisfaction. In this paper, we investigate how ccOPF feedback relates to the full-information hOPF. To this end, we introduce different dimensions of the price of uncertainty. Using mild assumptions on the uncertainty we present sufficient conditions when ccOPF is identical to hOPF. We suggest using the total variational distance of probability densities to quantify the performance gap of hOPF and ccOPF. Finally, we draw upon a tutorial example to illustrate our results.
4.2CRJul 26, 2024
Accurate and Scalable Detection and Investigation of Cyber Persistence ThreatsQi Liu, Muhammad Shoaib, Mati Ur Rehman et al.
In Advanced Persistent Threat (APT) attacks, achieving stealthy persistence within target systems is often crucial for an attacker's success. This persistence allows adversaries to maintain prolonged access, often evading detection mechanisms. Recognizing its pivotal role in the APT lifecycle, this paper introduces Cyber Persistence Detector (CPD), a novel system dedicated to detecting cyber persistence through provenance analytics. CPD is founded on the insight that persistent operations typically manifest in two phases: the "persistence setup" and the subsequent "persistence execution". By causally relating these phases, we enhance our ability to detect persistent threats. First, CPD discerns setups signaling an impending persistent threat and then traces processes linked to remote connections to identify persistence execution activities. A key feature of our system is the introduction of pseudo-dependency edges (pseudo-edges), which effectively connect these disjoint phases using data provenance analysis, and expert-guided edges, which enable faster tracing and reduced log size. These edges empower us to detect persistence threats accurately and efficiently. Moreover, we propose a novel alert triage algorithm that further reduces false positives associated with persistence threats. Evaluations conducted on well-known datasets demonstrate that our system reduces the average false positive rate by 93% compared to state-of-the-art methods.
ALDI++: Automatic and parameter-less discord and outlier detection for building energy load profilesMatias Quintana, Till Stoeckmann, June Young Park et al.
Data-driven building energy prediction is an integral part of the process for measurement and verification, building benchmarking, and building-to-grid interaction. The ASHRAE Great Energy Predictor III (GEPIII) machine learning competition used an extensive meter data set to crowdsource the most accurate machine learning workflow for whole building energy prediction. A significant component of the winning solutions was the pre-processing phase to remove anomalous training data. Contemporary pre-processing methods focus on filtering statistical threshold values or deep learning methods requiring training data and multiple hyper-parameters. A recent method named ALDI (Automated Load profile Discord Identification) managed to identify these discords using matrix profile, but the technique still requires user-defined parameters. We develop ALDI++, a method based on the previous work that bypasses user-defined parameters and takes advantage of discord similarity. We evaluate ALDI++ against a statistical threshold, variational auto-encoder, and the original ALDI as baselines in classifying discords and energy forecasting scenarios. Our results demonstrate that while the classification performance improvement over the original method is marginal, ALDI++ helps achieve the best forecasting error improving 6% over the winning's team approach with six times less computation time.
2.0LGFeb 6, 2023
ProbPNN: Enhancing Deep Probabilistic Forecasting with Statistical InformationBenedikt Heidrich, Kaleb Phipps, Oliver Neumann et al.
Probabilistic forecasts are essential for various downstream applications such as business development, traffic planning, and electrical grid balancing. Many of these probabilistic forecasts are performed on time series data that contain calendar-driven periodicities. However, existing probabilistic forecasting methods do not explicitly take these periodicities into account. Therefore, in the present paper, we introduce a deep learning-based method that considers these calendar-driven periodicities explicitly. The present paper, thus, has a twofold contribution: First, we apply statistical methods that use calendar-driven prior knowledge to create rolling statistics and combine them with neural networks to provide better probabilistic forecasts. Second, we benchmark ProbPNN with state-of-the-art benchmarks by comparing the achieved normalised continuous ranked probability score (nCRPS) and normalised Pinball Loss (nPL) on two data sets containing in total more than 1000 time series. The results of the benchmarks show that using statistical forecasting components improves the probabilistic forecast performance and that ProbPNN outperforms other deep learning forecasting methods whilst requiring less computation costs.
2.0LGFeb 3, 2023
Creating Probabilistic Forecasts from Arbitrary Deterministic Forecasts using Conditional Invertible Neural NetworksKaleb Phipps, Benedikt Heidrich, Marian Turowski et al.
In various applications, probabilistic forecasts are required to quantify the inherent uncertainty associated with the forecast. However, numerous modern forecasting methods are still designed to create deterministic forecasts. Transforming these deterministic forecasts into probabilistic forecasts is often challenging and based on numerous assumptions that may not hold in real-world situations. Therefore, the present article proposes a novel approach for creating probabilistic forecasts from arbitrary deterministic forecasts. In order to implement this approach, we use a conditional Invertible Neural Network (cINN). More specifically, we apply a cINN to learn the underlying distribution of the data and then combine the uncertainty from this distribution with an arbitrary deterministic forecast to generate accurate probabilistic forecasts. Our approach enables the simple creation of probabilistic forecasts without complicated statistical loss functions or further assumptions. Besides showing the mathematical validity of our approach, we empirically show that our approach noticeably outperforms traditional methods for including uncertainty in deterministic forecasts and generally outperforms state-of-the-art probabilistic forecasting benchmarks.
1.2SYMar 20, 2025
Privacy-Preserving Utilization of Distribution System Flexibility for Enhanced TSO-DSO Interoperability: A Novel Machine Learning-Based Optimal Power Flow ApproachBurak Dindar, Can Berk Saner, Hüseyin K. Çakmak et al.
Due to the transformation of the power system, the effective use of flexibility from the distribution system (DS) is becoming crucial for efficient network management. Leveraging this flexibility requires interoperability among stakeholders, including Transmission System Operators (TSOs) and Distribution System Operators (DSOs). However, data privacy concerns among stakeholders present significant challenges for utilizing this flexibility effectively. To address these challenges, we propose a machine learning (ML)-based method in which the technical constraints of the DSs are represented by ML models trained exclusively on non-sensitive data. Using these models, the TSO can solve the optimal power flow (OPF) problem and directly determine the dispatch of flexibility-providing units (FPUs), in our case, distributed generators (DGs), in a single round of communication. To achieve this, we introduce a novel neural network (NN) architecture specifically designed to efficiently represent the feasible region of the DSs, ensuring computational effectiveness. Furthermore, we incorporate various PQ charts rather than idealized ones, demonstrating that the proposed method is adaptable to a wide range of FPU characteristics. To assess the effectiveness of the proposed method, we benchmark it against the standard AC-OPF on multiple DSs with meshed connections and multiple points of common coupling (PCCs) with varying voltage magnitudes. The numerical results indicate that the proposed method achieves performant results while prioritizing data privacy. Additionally, since this method directly determines the dispatch of FPUs, it eliminates the need for an additional disaggregation step. By representing the DSs technical constraints through ML models trained exclusively on non-sensitive data, the transfer of sensitive information between stakeholders is prevented.
2.6LGSep 29, 2024
Generating peak-aware pseudo-measurements for low-voltage feeders using metadata of distribution system operatorsManuel Treutlein, Marc Schmidt, Roman Hahn et al.
Distribution system operators (DSOs) must cope with new challenges such as the reconstruction of distribution grids along climate neutrality pathways or the ability to manage and control consumption and generation in the grid. In order to meet the challenges, measurements within the distribution grid often form the basis for DSOs. Hence, it is an urgent problem that measurement devices are not installed in many low-voltage (LV) grids. In order to overcome this problem, we present an approach to estimate pseudo-measurements for non-measured LV feeders based on the metadata of the respective feeder using regression models. The feeder metadata comprise information about the number of grid connection points, the installed power of consumers and producers, and billing data in the downstream LV grid. Additionally, we use weather data, calendar data and timestamp information as model features. The existing measurements are used as model target. We extensively evaluate the estimated pseudo-measurements on a large real-world dataset with 2,323 LV feeders characterized by both consumption and feed-in. For this purpose, we introduce peak metrics inspired by the BigDEAL challenge for the peak magnitude, timing and shape for both consumption and feed-in. As regression models, we use XGBoost, a multilayer perceptron (MLP) and a linear regression (LR). We observe that XGBoost and MLP outperform the LR. Furthermore, the results show that the approach adapts to different weather, calendar and timestamp conditions and produces realistic load curves based on the feeder metadata. In the future, the approach can be adapted to other grid levels like substation transformers and can supplement research fields like load modeling, state estimation and LV load forecasting.
pyWATTS: Python Workflow Automation Tool for Time SeriesBenedikt Heidrich, Andreas Bartschat, Marian Turowski et al.
Time series data are fundamental for a variety of applications, ranging from financial markets to energy systems. Due to their importance, the number and complexity of tools and methods used for time series analysis is constantly increasing. However, due to unclear APIs and a lack of documentation, researchers struggle to integrate them into their research projects and replicate results. Additionally, in time series analysis there exist many repetitive tasks, which are often re-implemented for each project, unnecessarily costing time. To solve these problems we present \texttt{pyWATTS}, an open-source Python-based package that is a non-sequential workflow automation tool for the analysis of time series data. pyWATTS includes modules with clearly defined interfaces to enable seamless integration of new or existing methods, subpipelining to easily reproduce repetitive tasks, load and save functionality to simply replicate results, and native support for key Python machine learning libraries such as scikit-learn, PyTorch, and Keras.
2.6LGNov 30, 2024
AutoPQ: Automating Quantile estimation from Point forecasts in the context of sustainabilityStefan Meisenbacher, Kaleb Phipps, Oskar Taubert et al.
Optimizing smart grid operations relies on critical decision-making informed by uncertainty quantification, making probabilistic forecasting a vital tool. Designing such forecasting models involves three key challenges: accurate and unbiased uncertainty quantification, workload reduction for data scientists during the design process, and limitation of the environmental impact of model training. In order to address these challenges, we introduce AutoPQ, a novel method designed to automate and optimize probabilistic forecasting for smart grid applications. AutoPQ enhances forecast uncertainty quantification by generating quantile forecasts from an existing point forecast by using a conditional Invertible Neural Network (cINN). AutoPQ also automates the selection of the underlying point forecasting method and the optimization of hyperparameters, ensuring that the best model and configuration is chosen for each application. For flexible adaptation to various performance needs and available computing power, AutoPQ comes with a default and an advanced configuration, making it suitable for a wide range of smart grid applications. Additionally, AutoPQ provides transparency regarding the electricity consumption required for performance improvements. We show that AutoPQ outperforms state-of-the-art probabilistic forecasting methods while effectively limiting computational effort and hence environmental impact. Additionally and in the context of sustainability, we quantify the electricity consumption required for performance improvements.
3.6CRSep 26, 2025
A Global Analysis of Cyber Threats to the Energy Sector: "Currents of Conflict" from a Geopolitical PerspectiveGustavo Sánchez, Ghada Elbez, Veit Hagenmeyer
The escalating frequency and sophistication of cyber threats increased the need for their comprehensive understanding. This paper explores the intersection of geopolitical dynamics, cyber threat intelligence analysis, and advanced detection technologies, with a focus on the energy domain. We leverage generative artificial intelligence to extract and structure information from raw cyber threat descriptions, enabling enhanced analysis. By conducting a geopolitical comparison of threat actor origins and target regions across multiple databases, we provide insights into trends within the general threat landscape. Additionally, we evaluate the effectiveness of cybersecurity tools -- with particular emphasis on learning-based techniques -- in detecting indicators of compromise for energy-targeted attacks. This analysis yields new insights, providing actionable information to researchers, policy makers, and cybersecurity professionals.
9.4LGMar 3, 2025
Decision-Focused Fine-Tuning of Time Series Foundation Models for Dispatchable Feeder OptimizationMaximilian Beichter, Nils Friederich, Janik Pinter et al.
Time series foundation models provide a universal solution for generating forecasts to support optimization problems in energy systems. Those foundation models are typically trained in a prediction-focused manner to maximize forecast quality. In contrast, decision-focused learning directly improves the resulting value of the forecast in downstream optimization rather than merely maximizing forecasting quality. The practical integration of forecast values into forecasting models is challenging, particularly when addressing complex applications with diverse instances, such as buildings. This becomes even more complicated when instances possess specific characteristics that require instance-specific, tailored predictions to increase the forecast value. To tackle this challenge, we use decision-focused fine-tuning within time series foundation models to offer a scalable and efficient solution for decision-focused learning applied to the dispatchable feeder optimization problem. To obtain more robust predictions for scarce building data, we use Moirai as a state-of-the-art foundation model, which offers robust and generalized results with few-shot parameter-efficient fine-tuning. Comparing the decision-focused fine-tuned Moirai with a state-of-the-art classical prediction-focused fine-tuning Morai, we observe an improvement of 9.45% in average total daily costs.
2.6LGNov 30, 2024
On autoregressive deep learning models for day-ahead wind power forecasting with irregular shutdowns due to redispatchingStefan Meisenbacher, Silas Aaron Selzer, Mehdi Dado et al.
Renewable energies and their operation are becoming increasingly vital for the stability of electrical power grids since conventional power plants are progressively being displaced, and their contribution to redispatch interventions is thereby diminishing. In order to consider renewable energies like Wind Power (WP) for such interventions as a substitute, day-ahead forecasts are necessary to communicate their availability for redispatch planning. In this context, automated and scalable forecasting models are required for the deployment to thousands of locally-distributed onshore WP turbines. Furthermore, the irregular interventions into the WP generation capabilities due to redispatch shutdowns pose challenges in the design and operation of WP forecasting models. Since state-of-the-art forecasting methods consider past WP generation values alongside day-ahead weather forecasts, redispatch shutdowns may impact the forecast. Therefore, the present paper highlights these challenges and analyzes state-of-the-art forecasting methods on data sets with both regular and irregular shutdowns. Specifically, we compare the forecasting accuracy of three autoregressive Deep Learning (DL) methods to methods based on WP curve modeling. Interestingly, the latter achieve lower forecasting errors, have fewer requirements for data cleaning during modeling and operation while being computationally more efficient, suggesting their advantages in practical applications.
14.1LGFeb 3, 2022
Review of automated time series forecasting pipelinesStefan Meisenbacher, Marian Turowski, Kaleb Phipps et al.
Time series forecasting is fundamental for various use cases in different domains such as energy systems and economics. Creating a forecasting model for a specific use case requires an iterative and complex design process. The typical design process includes the five sections (1) data pre-processing, (2) feature engineering, (3) hyperparameter optimization, (4) forecasting method selection, and (5) forecast ensembling, which are commonly organized in a pipeline structure. One promising approach to handle the ever-growing demand for time series forecasts is automating this design process. The present paper, thus, analyzes the existing literature on automated time series forecasting pipelines to investigate how to automate the design process of forecasting models. Thereby, we consider both Automated Machine Learning (AutoML) and automated statistical forecasting methods in a single forecasting pipeline. For this purpose, we firstly present and compare the proposed automation methods for each pipeline section. Secondly, we analyze the automation methods regarding their interaction, combination, and coverage of the five pipeline sections. For both, we discuss the literature, identify problems, give recommendations, and suggest future research. This review reveals that the majority of papers only cover two or three of the five pipeline sections. We conclude that future research has to holistically consider the automation of the forecasting pipeline to enable the large-scale application of time series forecasting.
6.5LGOct 26, 2021
Concepts for Automated Machine Learning in Smart Grid ApplicationsStefan Meisenbacher, Janik Pinter, Tim Martin et al.
Undoubtedly, the increase of available data and competitive machine learning algorithms has boosted the popularity of data-driven modeling in energy systems. Applications are forecasts for renewable energy generation and energy consumption. Forecasts are elementary for sector coupling, where energy-consuming sectors are interconnected with the power-generating sector to address electricity storage challenges by adding flexibility to the power system. However, the large-scale application of machine learning methods in energy systems is impaired by the need for expert knowledge, which covers machine learning expertise and a profound understanding of the application's process. The process knowledge is required for the problem formalization, as well as the model validation and application. The machine learning skills include the processing steps of i) data pre-processing, ii) feature engineering, extraction, and selection, iii) algorithm selection, iv) hyperparameter optimization, and possibly v) post-processing of the model's output. Tailoring a model for a particular application requires selecting the data, designing various candidate models and organizing the data flow between the processing steps, selecting the most suitable model, and monitoring the model during operation - an iterative and time-consuming procedure. Automated design and operation of machine learning aim to reduce the human effort to address the increasing demand for data-driven models. We define five levels of automation for forecasting in alignment with the SAE standard for autonomous vehicles, where manual design and application reflect Automation level 0.
1.2DCFeb 18, 2020
A Scalable Method for Scheduling Distributed Energy Resources using Parallelized Population-based MetaheuristicsHatem Khalloof, Wilfried Jakob, Shadi Shahoud et al.
Recent years have seen an increasing integration of distributed renewable energy resources into existing electric power grids. Due to the uncertain nature of renewable energy resources, network operators are faced with new challenges in balancing load and generation. In order to meet the new requirements, intelligent distributed energy resource plants can be used which provide as virtual power plants e.g. demand side management or flexible generation. However, the calculation of an adequate schedule for the unit commitment of such distributed energy resources is a complex optimization problem which is typically too complex for standard optimization algorithms if large numbers of distributed energy resources are considered. For solving such complex optimization tasks, population-based metaheuristics -- as e.g. evolutionary algorithms -- represent powerful alternatives. Admittedly, evolutionary algorithms do require lots of computational power for solving such problems in a timely manner. One promising solution for this performance problem is the parallelization of the usually time-consuming evaluation of alternative solutions. In the present paper, a new generic and highly scalable parallel method for unit commitment of distributed energy resources using metaheuristic algorithms is presented. It is based on microservices, container virtualization and the publish/subscribe messaging paradigm for scheduling distributed energy resources. Scalability and applicability of the proposed solution are evaluated by performing parallelized optimizations in a big data environment for three distinct distributed energy resource scheduling scenarios. The new method provides cluster or cloud parallelizability and is able to deal with a comparably large number of distributed energy resources. The application of the new proposed method results in very good performance for scaling up optimization speed.
1.8LGMar 18, 2019
Probabilistic Energy Forecasting using Quantile Regressions based on a new Nearest Neighbors Quantile FilterJorge Ángel González Ordiano, Lutz Gröll, Ralf Mikut et al.
Parametric quantile regressions are a useful tool for creating probabilistic energy forecasts. Nonetheless, since classical quantile regressions are trained using a non-differentiable cost function, their creation using complex data mining techniques (e.g., artificial neural networks) may be complicated. This article presents a method that uses a new nearest neighbors quantile filter to obtain quantile regressions independently of the utilized data mining technique and without the non-differentiable cost function. Thereafter, a validation of the presented method using the dataset of the Global Energy Forecasting Competition of 2014 is undertaken. The results show that the presented method is able to solve the competition's task with a similar accuracy and in a similar time as the competition's winner, but requiring a much less powerful computer. This property may be relevant in an online forecasting service for which the fast computation of probabilistic forecasts using not so powerful machines is required.
1.2SYApr 3, 2019
Chance-Constrained AC Optimal Power Flow -- A Polynomial Chaos ApproachT. Mühlpfordt, L. Roald, V. Hagenmeyer et al.
As the share of renewables in the grid increases, the operation of power systems becomes more challenging. The present paper proposes a method to formulate and solve chance-constrained optimal power flow while explicitly considering the full nonlinear AC power flow equations and stochastic uncertainties. We use polynomial chaos expansion to model the effects of arbitrary uncertainties of finite variance, which enables to predict and optimize the system state for a range of operating conditions. We apply chance constraints to limit the probability of violations of inequality constraints. Our method incorporates a more detailed and a more flexible description of both the controllable variables and the resulting system state than previous methods. Two case studies highlight the efficacy of the method, with a focus on satisfaction of the AC power flow equations and on the accurate computation of moments of all random variables.