Lang Tong

Liyan Jia, Jinsub Kim, Robert J. Thomas et al.

The problem of characterizing impacts of data quality on real-time locational marginal price (LMP) is considered. Because the real-time LMP is computed from the estimated network topology and system state, bad data that cause errors in topology processing and state estimation affect real-time LMP. It is shown that the power system state space is partitioned into price regions of convex polytopes. Under different bad data models, the worst case impacts of bad data on real-time LMP are analyzed. Numerical simulations are used to illustrate worst case performance for IEEE-14 and IEEE-118 networks.

Yuting Ji, Robert J. Thomas, Lang Tong

The short-term forecasting of real-time locational marginal price (LMP) and network congestion is considered from a system operator perspective. A new probabilistic forecasting technique is proposed based on a multiparametric programming formulation that partitions the uncertainty parameter space into critical regions from which the conditional probability distribution of the real-time LMP/congestion is obtained. The proposed method incorporates load/generation forecast, time varying operation constraints, and contingency models. By shifting the computation cost associated with multiparametric programs offline, the online computation cost is significantly reduced. An online simulation technique by generating critical regions dynamically is also proposed, which results in several orders of magnitude improvement in the computational cost over standard Monte Carlo methods.

Yuting Ji, Tongxin Zheng, Lang Tong

The problem of multi-area interchange scheduling in the presence of stochastic generation and load is considered. A new interchange scheduling technique based on a two-stage stochastic minimization of overall expected operating cost is proposed. Because directly solving the stochastic optimization is intractable, an equivalent problem that maximizes the expected social welfare is formulated. The proposed technique leverages the operator's capability of forecasting locational marginal prices (LMPs) and obtains the optimal interchange schedule without iterations among operators.

Shiyao Chen, Lang Tong, Qing Zhao

The problem of minimizing queueing delay of opportunistic access of multiple continuous time Markov channels is considered. A new access policy based on myopic sensing and adaptive transmission (MS-AT) is proposed. Under the framework of risk sensitive constrained Markov decision process with effective bandwidth as a measure of queueing delay, it is shown that MS-AT achieves simultaneously throughput and delay optimality. It is shown further that both the effective bandwidth and the throughput of MS-AT are two-segment piece-wise linear functions of the collision constraint (maximum allowable conditional collision probability) with the effective bandwidth and throughput coinciding in the regime of tight collision constraints. Analytical and simulations comparisons with the myopic sensing and memoryless transmission (MS-MT) policy which is throughput optimal but delay suboptimal in the regime of tight collision constraints.

Siying Li, Lang Tong, Timothy Mount et al.

We study the optimal green hydrogen production and energy market participation of a renewable-colocated hydrogen producer (RCHP) that utilizes onsite renewable generation for both hydrogen production and grid services. Under deterministic and stochastic profit-maximization frameworks, we analyze RCHP's multiple market participation models and derive closed-form optimal scheduling policies that dynamically allocate renewable energy to hydrogen production and electricity export to the wholesale market. Analytical characterizations of the RCHP's operating profit and the optimal sizing of renewable and electrolyzer capacities are obtained. We use real-time renewable generation and electricity price data from three independent system operators to evaluate the impacts of market prices and environmental policies on RCHP's profitability.

Cong Chen, Ahmed S. Alahmed, Timothy D. Mount et al.

We consider the aggregation of distributed energy resources (DERs), such as solar PV, energy storage, and flexible loads, by a profit-seeking aggregator participating directly in the wholesale market under distribution network access constraints. We propose a competitive DER aggregator (DERA) model that directly controls local DERs to maximize its profits, while ensuring each aggregated customer gains a surplus higher than their surplus under the regulated retail tariff. The DERA participates in the wholesale electricity market as virtual storage with optimized generation offers and consumption bids derived from the propoed competitive aggregation model. Also derived are DERA's bid curves for the distribution network access and DERA's profitability when competing with the regulated retail tariff. We show that, with the same distribution network access, the proposed DERA's wholesale market participation achieves the same welfare-maximizing outcome as when its customers participate directly in the wholesale market. Extensive numerical studies compare the proposed DERA with existing methods in terms of customer surplus and DERA profit. We empirically evaluate how many DERAs can survive in the competition at long-run equilibrium, and assess the impacts of DER adoption levels and distribution network access on short-run operations.

Xinyi Wang, Meijen Lee, Qing Zhao et al.

Probabilistic time series forecasting predicts the conditional probability distributions of the time series at a future time given past realizations. Such techniques are critical in risk-based decision-making and planning under uncertainties. Existing approaches are primarily based on parametric or semi-parametric time-series models that are restrictive, difficult to validate, and challenging to adapt to varying conditions. This paper proposes a nonparametric method based on the classic notion of {\em innovations} pioneered by Norbert Wiener and Gopinath Kallianpur that causally transforms a nonparametric random process to an independent and identical uniformly distributed {\em innovations process}. We present a machine-learning architecture and a learning algorithm that circumvent two limitations of the original Wiener-Kallianpur innovations representation: (i) the need for known probability distributions of the time series and (ii) the existence of a causal decoder that reproduces the original time series from the innovations representation. We develop a deep-learning approach and a Monte Carlo sampling technique to obtain a generative model for the predicted conditional probability distribution of the time series based on a weak notion of Wiener-Kallianpur innovations representation. The efficacy of the proposed probabilistic forecasting technique is demonstrated on a variety of electricity price datasets, showing marked improvement over leading benchmarks of probabilistic forecasting techniques.

Xinyi Wang, Mei-jen Lee, Qing Zhao et al.

We consider novelty detection in time series with unknown and nonparametric probability structures. A deep learning approach is proposed to causally extract an innovations sequence consisting of novelty samples statistically independent of all past samples of the time series. A novelty detection algorithm is developed for the online detection of novel changes in the probability structure in the innovations sequence. A minimax optimality under a Bayes risk measure is established for the proposed novelty detection method, and its robustness and efficacy are demonstrated in experiments using real and synthetic datasets.

Minjae Jeon, Lang Tong, Qing Zhao

We investigate the problem of serving deferrable and nondeferrable electric demands with colocated stochastic supply and grid-imported electricity. Deferrable demands arrive randomly and can be delayed within their service deadlines. Nondeferrable demands are always present and must be served immediately, but the quantity served depends on the cost of electricity. Colocated supply is stochastic with zero marginal cost. It can be used to meet demand or exported to the grid to maximize profit. The stochasticity of demands and local supply makes optimal scheduling a Markov decision process with continuous (uncountable) state and action spaces. Under deterministic, time-varying, and piecewise-linear retail pricing of electricity, we show that the optimal demand scheduling follows the {\em Principle of Procrastination}, which reduces the infinite-dimensional policy space to a finite-dimensional Euclidean space defined by three procrastination parameters for each deferrable demand. For settings in which the underlying probability distributions are unknown, we propose a {\em Procrastination Threshold Reinforcement Learning} algorithm. Numerical experiments based on real-world test data confirm that the proposed threshold learning algorithm closely approximates the optimal policy and outperforms standard benchmarks.

Xinyi Wang, Lang Tong, Qing Zhao

Generative probabilistic forecasting produces future time series samples according to the conditional probability distribution given past time series observations. Such techniques are essential in risk-based decision-making and planning under uncertainty with broad applications in grid operations, including electricity price forecasting, risk-based economic dispatch, and stochastic optimizations. Inspired by Wiener and Kallianpur's innovation representation, we propose a weak innovation autoencoder architecture and a learning algorithm to extract independent and identically distributed innovation sequences from nonparametric stationary time series. We show that the weak innovation sequence is Bayesian sufficient, which makes the proposed weak innovation autoencoder a canonical architecture for generative probabilistic forecasting. The proposed technique is applied to forecasting highly volatile real-time electricity prices, demonstrating superior performance across multiple forecasting measures over leading probabilistic and point forecasting techniques.

Lang Tong, Xinyi Wang, Qing Zhao

Purpose:This article advocates for the development of a next-generation grid monitoring and control system designed for future grids dominated by inverter-based resources. Leveraging recent progress in generative artificial intelligence (AI), machine learning, and networking technology, we develop a physics-based AI foundation model with high-resolution synchro-waveform measurement technology to enhance grid resilience and reduce economic losses from outages. Methods and Results:The proposed framework adopts the AI Foundation Model paradigm, where a generative and pre-trained (GPT) foundation model extracts physical features from power system measurements, enabling adaptation to a wide range of grid operation tasks. Replacing the large language models used in popular AI foundation models, this approach is based on the Wiener-Kallianpur-Rosenblatt innovation model for power system time series, trained to capture the physical laws of power flows and sinusoidal characteristics of grid measurements. The pre-trained foundation model causally extracts sufficient statistics from grid measurement time series for various downstream applications, including anomaly detection, over-current protection, probabilistic forecasting, and data compression for streaming synchro-waveform data. Numerical simulations using field-collected data demonstrate significantly improved fault detection accuracy and detection speed. Conclusion:The future grid will be rich in inverter-based resources, making it highly dynamic, stochastic, and low inertia. This work underscores the limitations of existing Supervisory-Control-and-Data-Acquisition and Phasor-Measurement-Unit monitoring systems and advocates for AI-enabled monitoring and control with high-resolution synchro-waveform technology to provide accurate situational awareness, rapid response to faults, and robust network protection.

Lang Tong, Xinyi Wang

This paper introduces an Artificial Intelligence (AI) foundation model for time series in engineering applications, where causal operations are required for real-time monitoring and control. Since engineering time series are governed by physical, rather than linguistic, laws, large-language-model-based AI foundation models may be ineffective or inefficient. Building on the classical innovations representation theory of Wiener, Kallianpur, and Rosenblatt, we propose Time Series GPT (TS-GPT) -- an innovations-representation-based Generative Pre-trained Transformer for engineering monitoring and control. As an example of foundation model adaptation, we consider Probabilistic Generative Forecasting, which produces future time series samples from conditional probability distributions given past realizations. We demonstrate the effectiveness of TS-GPT in forecasting real-time locational marginal prices using historical data from U.S. independent system operators.

Siying Li, Lang Tong, Timothy D. Mount

We develop an energy management system (EMS) for artificial intelligence (AI) data centers with colocated renewable generation. Under a cost-minimizing framework, the EMS of renewable-colocated data center (RCDC) co-optimizes AI workload scheduling, on-site renewable utilization, and electricity market participation. Within both wholesale and retail market participation models, the economic benefit of the RCDC operation is maximized. Empirical evaluations using real-world traces of electricity prices, data center power consumption, and renewable generation demonstrate significant electricity cost reduction from renewable and AI data center colocations.

Xinyi Wang, Qing Zhao, Lang Tong

This paper introduces a generative AI approach to probabilistic forecasting of real-time electricity market signals, including locational marginal prices, interregional price spreads, and demand-supply imbalances. We present WIAE-GPF, a Weak Innovation AutoEncoder-based Generative Probabilistic Forecasting architecture that generates future samples of multivariate time series. Unlike traditional black-box models, WIAE-GPF offers interpretability through the Wiener-Kallianpur innovation representation for nonparametric time series, making it a nonparametric generalization of the Wiener/Kalman filter-based forecasting. A novel learning algorithm with structural convergence guarantees is proposed, ensuring that, under ideal training conditions, the generated forecast samples match the ground truth conditional probability distribution. Extensive tests using publicly available data from U.S. independent system operators under various point and probabilistic forecasting metrics demonstrate that WIAE-GPF consistently outperforms classical methods and cutting-edge machine learning techniques.

Sudeep Salgia, Qing Zhao, Lang Tong

We consider the problem of uniformity testing of Lipschitz continuous distributions with bounded support. The alternative hypothesis is a composite set of Lipschitz continuous distributions that are at least $\varepsilon$ away in $\ell_1$ distance from the uniform distribution. We propose a sequential test that adapts to the unknown distribution under the alternative hypothesis. Referred to as the Adaptive Binning Coincidence (ABC) test, the proposed strategy adapts in two ways. First, it partitions the set of alternative distributions into layers based on their distances to the uniform distribution. It then sequentially eliminates the alternative distributions layer by layer in decreasing distance to the uniform, and subsequently takes advantage of favorable situations of a distant alternative by exiting early. Second, it adapts, across layers of the alternative distributions, the resolution level of the discretization for computing the coincidence statistic. The farther away the layer is from the uniform, the coarser the discretization is needed for eliminating/exiting this layer. It thus exits both early in the detection process and quickly by using a lower resolution to take advantage of favorable alternative distributions. The ABC test builds on a novel sequential coincidence test for discrete distributions, which is of independent interest. We establish the sample complexity of the proposed tests as well as a lower bound.

Xinyi Wang, Lang Tong

An innovations sequence of a time series is a sequence of independent and identically distributed random variables with which the original time series has a causal representation. The innovation at a time is statistically independent of the history of the time series. As such, it represents the new information contained at present but not in the past. Because of its simple probability structure, an innovations sequence is the most efficient signature of the original. Unlike the principle or independent component analysis representations, an innovations sequence preserves not only the complete statistical properties but also the temporal order of the original time series. An long-standing open problem is to find a computationally tractable way to extract an innovations sequence of non-Gaussian processes. This paper presents a deep learning approach, referred to as Innovations Autoencoder (IAE), that extracts innovations sequences using a causal convolutional neural network. An application of IAE to the one-class anomalous sequence detection problem with unknown anomaly and anomaly-free models is also presented.

Behrouz Azimian, Reetam Sen Biswas, Shiva Moshtagh et al.

Time-synchronized state estimation for reconfigurable distribution networks is challenging because of limited real-time observability. This paper addresses this challenge by formulating a deep learning (DL)-based approach for topology identification (TI) and unbalanced three-phase distribution system state estimation (DSSE). Two deep neural networks (DNNs) are trained for time-synchronized DNN-based TI and DSSE, respectively, for systems that are incompletely observed by synchrophasor measurement devices (SMDs) in real-time. A data-driven approach for judicious SMD placement to facilitate reliable TI and DSSE is also provided. Robustness of the proposed methodology is demonstrated by considering non-Gaussian noise in the SMD measurements. A comparison of the DNN-based DSSE with more conventional approaches indicates that the DL-based approach gives better accuracy with smaller number of SMDs.

Kursat Rasim Mestav, Xinyi Wang, Lang Tong

A deep learning approach is proposed to detect data and system anomalies using high-resolution continuous point-on-wave (CPOW) or phasor measurements. Both the anomaly and anomaly-free measurement models are assumed to have unknown temporal dependencies and probability distributions. Historical training samples are assumed for the anomaly-free model, while no training samples are available for the anomaly measurements. By transforming the anomaly-free observations into uniform independent and identically distributed sequences via a generative adversarial network, the proposed approach deploys a uniformity test for anomaly detection at the sensor level. A distributed detection scheme that combines sensor level detections at the control center is also proposed that combines local detections to form more reliable detections. Numerical results demonstrate significant improvement over the state-of-the-art solutions for various bad-data cases using real and synthetic CPOW and PMU data sets.

Behrouz Azimian, Reetam Sen Biswas, Anamitra Pal et al.

Time-synchronized state estimation is a challenge for distribution systems because of limited real-time observability. This paper addresses this challenge by formulating a deep learning (DL)-based approach to perform unbalanced three-phase distribution system state estimation (DSSE). Initially, a data-driven approach for judicious measurement selection to facilitate reliable state estimation is provided. Then, a deep neural network (DNN) is trained to perform DSSE for systems that are incompletely observed by synchrophasor measurement devices (SMDs). Robustness of the proposed methodology is demonstrated by considering realistic measurement error models for SMDs. A comparative study of the DNN-based DSSE with classical linear state estimation indicates that the DL-based approach gives better accuracy with a significantly smaller number of SMDs.

Kursat Rasim Mestav, Lang Tong

The problem of detecting data anomaly is considered. Under the null hypothesis that models anomaly-free data, measurements are assumed to be from an unknown distribution with some authenticated historical samples. Under the composite alternative hypothesis, measurements are from an unknown distribution positive distance away from the distribution under the null hypothesis. No training data are available for the distribution of anomaly data. A semi-supervised deep learning technique based on an inverse generative adversary network is proposed.

Kursat Rasim Mestav, Jaime Luengo-Rozas, Lang Tong

The problem of state estimation for unobservable distribution systems is considered. A deep learning approach to Bayesian state estimation is proposed for real-time applications. The proposed technique consists of distribution learning of stochastic power injection, a Monte Carlo technique for the training of a deep neural network for state estimation, and a Bayesian bad-data detection and filtering algorithm. Structural characteristics of the deep neural networks are investigated. Simulations illustrate the accuracy of Bayesian state estimation for unobservable systems and demonstrate the benefit of employing a deep neural network. Numerical results show the robustness of Bayesian state estimation against modeling and estimation errors and the presence of bad and missing data. Comparing with pseudo-measurement techniques, direct Bayesian state estimation via deep learning neural network outperforms existing benchmarks.

Sevi Baltaoglu, Lang Tong, Qing Zhao

We consider the problem of optimal bidding for virtual trading in two-settlement electricity markets. A virtual trader aims to arbitrage on the differences between day-ahead and real-time market prices; both prices, however, are random and unknown to market participants. An online learning algorithm is proposed to maximize the cumulative payoff over a finite number of trading sessions by allocating the trader's budget among his bids for K options in each session. It is shown that the proposed algorithm converges, with an almost optimal convergence rate, to the global optimal corresponding to the case when the underlying price distribution is known. The proposed algorithm is also generalized for trading strategies with a risk measure. By using both cumulative payoff and Sharpe ratio as performance metrics, evaluations were performed based on historical data spanning ten year period of NYISO and PJM markets. It was shown that the proposed strategy outperforms standard benchmarks and the S&P 500 index over the same period.

Sevi Baltaoglu, Lang Tong, Qing Zhao

We study the online learning problem of a bidder who participates in repeated auctions. With the goal of maximizing his T-period payoff, the bidder determines the optimal allocation of his budget among his bids for $K$ goods at each period. As a bidding strategy, we propose a polynomial-time algorithm, inspired by the dynamic programming approach to the knapsack problem. The proposed algorithm, referred to as dynamic programming on discrete set (DPDS), achieves a regret order of $O(\sqrt{T\log{T}})$. By showing that the regret is lower bounded by $Ω(\sqrt{T})$ for any strategy, we conclude that DPDS is order optimal up to a $\sqrt{\log{T}}$ term. We evaluate the performance of DPDS empirically in the context of virtual trading in wholesale electricity markets by using historical data from the New York market. Empirical results show that DPDS consistently outperforms benchmark heuristic methods that are derived from machine learning and online learning approaches.

Weisi Deng, Yuting Ji, Lang Tong

The problem of probabilistic forecasting and online simulation of real-time electricity market with stochastic generation and demand is considered. By exploiting the parametric structure of the direct current optimal power flow, a new technique based on online dictionary learning (ODL) is proposed. The ODL approach incorporates real-time measurements and historical traces to produce forecasts of joint and marginal probability distributions of future locational marginal prices, power flows, and dispatch levels, conditional on the system state at the time of forecasting. Compared with standard Monte Carlo simulation techniques, the ODL approach offers several orders of magnitude improvement in computation time, making it feasible for online forecasting of market operations. Numerical simulations on large and moderate size power systems illustrate its performance and complexity features and its potential as a tool for system operators.

Jinsub Kim, Lang Tong, Robert J. Thomas

Data attacks on state estimation modify part of system measurements such that the tempered measurements cause incorrect system state estimates. Attack techniques proposed in the literature often require detailed knowledge of system parameters. Such information is difficult to acquire in practice. The subspace methods presented in this paper, on the other hand, learn the system operating subspace from measurements and launch attacks accordingly. Conditions for the existence of an unobservable subspace attack are obtained under the full and partial measurement models. Using the estimated system subspace, two attack strategies are presented. The first strategy aims to affect the system state directly by hiding the attack vector in the system subspace. The second strategy misleads the bad data detection mechanism so that data not under attack are removed. Performance of these attacks are evaluated using the IEEE 14-bus network and the IEEE 118-bus network.

Jinsub Kim, Lang Tong, Robert J. Thomas

A new mechanism aimed at misleading a power system control center about the source of a data attack is proposed. As a man-in-the-middle state attack, a data framing attack is proposed to exploit the bad data detection and identification mechanisms currently in use at most control centers. In particular, the proposed attack frames meters that are providing correct data as sources of bad data such that the control center will remove useful measurements that would otherwise be used by the state estimator. The optimal design of a data framing attack is formulated as a quadratically constrained quadratic program (QCQP). It is shown that the proposed attack is capable of perturbing the power system state estimate by an arbitrary degree controlling only half of a critical set of measurements that are needed to make a system unobservable. Implications of this attack on power system operations are discussed, and the attack performance is evaluated using benchmark systems.

Brandon Jones, Mark Campbell, Lang Tong

The fusion of independently obtained stochastic maps by collaborating mobile agents is considered. The proposed approach includes two parts: matching of stochastic maps and maximum likelihood alignment. In particular, an affine invariant hypergraph is constructed for each stochastic map, and a bipartite matching via a linear program is used to establish landmark correspondence between stochastic maps. A maximum likelihood alignment procedure is proposed to determine rotation and translation between common landmarks in order to construct a global map within a common frame of reference. A main feature of the proposed approach is its scalability with respect to the number of landmarks: the matching step has polynomial complexity and the maximum likelihood alignment is obtained in closed form. Experimental validation of the proposed fusion approach is performed using the Victoria Park benchmark dataset.