Daniel E. Quevedo

Alex S. Leong, Arunselvan Ramaswamy, Daniel E. Quevedo et al.

In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios.

Zihuai Zhao, Wanchun Liu, Daniel E. Quevedo et al.

Wireless networked control system (WNCS) connecting sensors, controllers, and actuators via wireless communications is a key enabling technology for highly scalable and low-cost deployment of control systems in the Industry 4.0 era. Despite the tight interaction of control and communications in WNCSs, most existing works adopt separative design approaches. This is mainly because the co-design of control-communication policies requires large and hybrid state and action spaces, making the optimal problem mathematically intractable and difficult to be solved effectively by classic algorithms. In this paper, we systematically investigate deep learning (DL)-based estimator-control-scheduler co-design for a model-unknown nonlinear WNCS over wireless fading channels. In particular, we propose a co-design framework with the awareness of the sensor's age-of-information (AoI) states and dynamic channel states. We propose a novel deep reinforcement learning (DRL)-based algorithm for controller and scheduler optimization utilizing both model-free and model-based data. An AoI-based importance sampling algorithm that takes into account the data accuracy is proposed for enhancing learning efficiency. We also develop novel schemes for enhancing the stability of joint training. Extensive experiments demonstrate that the proposed joint training algorithm can effectively solve the estimation-control-scheduling co-design problem in various scenarios and provide significant performance gain compared to separative design and some benchmark policies.

Jiazheng Chen, Wanchun Liu, Daniel E. Quevedo et al.

Remote state estimation of large-scale distributed dynamic processes plays an important role in Industry 4.0 applications. In this paper, we focus on the transmission scheduling problem of a remote estimation system. First, we derive some structural properties of the optimal sensor scheduling policy over fading channels. Then, building on these theoretical guidelines, we develop a structure-enhanced deep reinforcement learning (DRL) framework for optimal scheduling of the system to achieve the minimum overall estimation mean-square error (MSE). In particular, we propose a structure-enhanced action selection method, which tends to select actions that obey the policy structure. This explores the action space more effectively and enhances the learning efficiency of DRL agents. Furthermore, we introduce a structure-enhanced loss function to add penalties to actions that do not follow the policy structure. The new loss function guides the DRL to converge to the optimal policy structure quickly. Our numerical experiments illustrate that the proposed structure-enhanced DRL algorithms can save the training time by 50% and reduce the remote estimation MSE by 10% to 25% when compared to benchmark DRL algorithms. In addition, we show that the derived structural properties exist in a wide range of dynamic scheduling problems that go beyond remote state estimation.

Burak Demirel, Euhanna Ghadimi, Daniel E. Quevedo et al.

We consider a finite-horizon linear-quadratic optimal control problem where only a limited number of control messages are allowed for sending from the controller to the actuator. To restrict the number of control actions computed and transmitted by the controller, we employ a threshold-based event-triggering mechanism that decides whether or not a control message needs to be calculated and delivered. Due to the nature of threshold-based event-triggering algorithms, finding the optimal control sequence requires minimizing a quadratic cost function over a non-convex domain. In this paper, we firstly provide an exact solution to the non-convex problem mentioned above by solving an exponential number of quadratic programs. To reduce computational complexity, we, then, propose two efficient heuristic algorithms based on greedy search and the Alternating Direction Method of Multipliers (ADMM) method. Later, we consider a receding horizon control strategy for linear systems controlled by event-triggered controllers, and we also provide a complete stability analysis of receding horizon control that uses finite horizon optimization in the proposed class. Numerical examples testify to the viability of the presented design technique.

Alex S. Leong, Daniel E. Quevedo, Daniel Dolz et al.

This paper studies remote state estimation in the presence of an eavesdropper. A sensor transmits local state estimates over a packet dropping link to a remote estimator, while an eavesdropper can successfully overhear each sensor transmission with a certain probability. The objective is to determine when the sensor should transmit, in order to minimize the estimation error covariance at the remote estimator, while trying to keep the eavesdropper error covariance above a certain level. This is done by solving an optimization problem that minimizes a linear combination of the expected estimation error covariance and the negative of the expected eavesdropper error covariance. Structural results on the optimal transmission policy are derived, and shown to exhibit thresholding behaviour in the estimation error covariances. In the infinite horizon situation, it is shown that with unstable systems one can keep the expected estimation error covariance bounded while the expected eavesdropper error covariance becomes unbounded. An alternative measure of security, constraining the amount of information revealed to the eavesdropper, is also considered, and similar structural results on the optimal transmission policy are derived. In the infinite horizon situation with unstable systems, it is now shown that for any transmission policy which keeps the expected estimation error covariance bounded, the expected amount of information revealed to the eavesdropper is always lower bounded away from zero. An extension of our results to the transmission of measurements is also presented.

Lingying Huang, Alex S. Leong, Daniel E. Quevedo et al.

In this paper, we consider a remote state estimation problem in the presence of an eavesdropper. A smart sensor takes measurement of a discrete linear time-invariant (LTI) process and sends its local state estimate through a wireless network to a remote estimator. An eavesdropper can overhear the sensor transmissions with a certain probability. To enhance the system privacy level, we propose a novel encryption strategy to minimize a linear combination of the expected error covariance at the remote estimator and the negative of the expected error covariance at the eavesdropper, taking into account the cost of the encryption process. We prove the existence of an optimal deterministic and Markovian policy for such an encryption strategy over a finite time horizon. Two situations, namely, with or without knowledge of the eavesdropper estimation error covariance are studied and the optimal schedule is shown to satisfy the threshold-like structure in both cases. Numerical examples are given to illustrate the results.

Burak Demirel, Vijay Gupta, Daniel E. Quevedo et al.

We consider a stochastic system where the communication between the controller and the actuator is triggered by a threshold-based rule. The communication is performed across an unreliable link that stochastically erases transmitted packets. To decrease the communication burden, and as a partial protection against dropped packets, the controller sends a sequence of control commands to the actuator in each packet. These commands are stored in a buffer and applied sequentially until the next control packet arrives. In this context, we study dead-beat control laws and compute the expected linear-quadratic loss of the closed-loop system for any given event-threshold. Furthermore, we provide analytical expressions that quantify the trade-off between the communication cost and the control performance of event-triggered control systems. Numerical examples demonstrate the effectiveness of the proposed framework.

Jiazheng Chen, Wanchun Liu, Daniel E. Quevedo et al.

Remote state estimation of large-scale distributed dynamic processes plays an important role in Industry 4.0 applications. In this paper, by leveraging the theoretical results of structural properties of optimal scheduling policies, we develop a structure-enhanced deep reinforcement learning (DRL) framework for optimal scheduling of a multi-sensor remote estimation system to achieve the minimum overall estimation mean-square error (MSE). In particular, we propose a structure-enhanced action selection method, which tends to select actions that obey the policy structure. This explores the action space more effectively and enhances the learning efficiency of DRL agents. Furthermore, we introduce a structure-enhanced loss function to add penalty to actions that do not follow the policy structure. The new loss function guides the DRL to converge to the optimal policy structure quickly. Our numerical results show that the proposed structure-enhanced DRL algorithms can save the training time by 50% and reduce the remote estimation MSE by 10% to 25%, when compared to benchmark DRL algorithms.

Yuqing Ni, Alex S. Leong, Daniel E. Quevedo et al.

We consider pricing and selection with fading channels in a Stackelberg game framework. A channel server decides the channel prices and a client chooses which channel to use based on the remote estimation quality. We prove the existence of an optimal deterministic and Markovian policy for the client, and show that the optimal policies of both the server and the client have threshold structures when the time horizon is finite. Value iteration algorithm is applied to obtain the optimal solutions for both the server and client, and numerical simulations and examples are given to demonstrate the developed result.

Yuxing Zhong, Yuzhe Li, Daniel E. Quevedo et al.

This paper considers the optimal power allocation of a jamming attacker against remote state estimation. The attacker is self-sustainable and can harvest energy from the environment to launch attacks. The objective is to carefully allocate its attack power to maximize the estimation error at the fusion center. Regarding the attacker's knowledge of the system, two cases are discussed: (i) perfect channel knowledge and (ii) unknown channel model. For both cases, we formulate the problem as a Markov decision process (MDP) and prove the existence of an optimal deterministic and stationary policy. Moreover, for both cases, we develop algorithms to compute the allocation policy and demonstrate that the proposed algorithms for both cases converge to the optimal policy as time goes to infinity. Additionally, the optimal policy exhibits certain structural properties that can be leveraged to accelerate both algorithms. Numerical examples are given to illustrate the main results.

Yingjie Mi, Zihao Ren, Lei Wang et al.

This paper studies quantum-encrypted explicit MPC for constrained discrete-time linear systems in a cloud-based architecture. A finite-horizon quadratic MPC problem is solved offline to obtain a piecewise-affine controller. Shared quantum keys generated from Bell pairs and protected by quantum key distribution are used to encrypt the online control evaluation between the sensor and actuator. Based on this architecture, we develop a lightweight encrypted explicit MPC protocol, prove exact recovery of the plaintext control action, and characterize its computational efficiency. Numerical results demonstrate lower online complexity than classical encrypted MPC, while security is discussed in terms of confidentiality of plant data and control inputs.

Gaoyang Pang, Wanchun Liu, Dusit Niyato et al.

Wireless Human-Machine Collaboration (WHMC) represents a critical advancement for Industry 5.0, enabling seamless interaction between humans and machines across geographically distributed systems. As the WHMC systems become increasingly important for achieving complex collaborative control tasks, ensuring their stability is essential for practical deployment and long-term operation. Stability analysis certifies how the closed-loop system will behave under model randomness, which is essential for systems operating with wireless communications. However, the fundamental stability analysis of the WHMC systems remains an unexplored challenge due to the intricate interplay between the stochastic nature of wireless communications, dynamic human operations, and the inherent complexities of control system dynamics. This paper establishes a fundamental WHMC model incorporating dual wireless loops for machine and human control. Our framework accounts for practical factors such as short-packet transmissions, fading channels, and advanced HARQ schemes. We model human control lag as a Markov process, which is crucial for capturing the stochastic nature of human interactions. Building on this model, we propose a stochastic cycle-cost-based approach to derive a stability condition for the WHMC system, expressed in terms of wireless channel statistics, human dynamics, and control parameters. Our findings are validated through extensive numerical simulations and a proof-of-concept experiment, where we developed and tested a novel wireless collaborative cart-pole control system. The results confirm the effectiveness of our approach and provide a robust framework for future research on WHMC systems in more complex environments.

Jiazheng Chen, Wanchun Liu, Daniel Quevedo et al.

For cyber-physical systems in the 6G era, semantic communications connecting distributed devices for dynamic control and remote state estimation are required to guarantee application-level performance, not merely focus on communication-centric performance. Semantics here is a measure of the usefulness of information transmissions. Semantic-aware transmission scheduling of a large system often involves a large decision-making space, and the optimal policy cannot be obtained by existing algorithms effectively. In this paper, we first investigate the fundamental properties of the optimal semantic-aware scheduling policy and then develop advanced deep reinforcement learning (DRL) algorithms by leveraging the theoretical guidelines. Our numerical results show that the proposed algorithms can substantially reduce training time and enhance training performance compared to benchmark algorithms.

Gaoyang Pang, Kang Huang, Daniel E. Quevedo et al.

We consider a joint uplink and downlink scheduling problem of a fully distributed wireless networked control system (WNCS) with a limited number of frequency channels. Using elements of stochastic systems theory, we derive a sufficient stability condition of the WNCS, which is stated in terms of both the control and communication system parameters. Once the condition is satisfied, there exists a stationary and deterministic scheduling policy that can stabilize all plants of the WNCS. By analyzing and representing the per-step cost function of the WNCS in terms of a finite-length countable vector state, we formulate the optimal transmission scheduling problem into a Markov decision process and develop a deep reinforcement learning (DRL) based framework for solving it. To tackle the challenges of a large action space in DRL, we propose novel action space reduction and action embedding methods for the DRL framework that can be applied to various algorithms, including Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic Policy Gradient (TD3). Numerical results show that the proposed algorithm significantly outperforms benchmark policies.

M. Schulze Darup, A. B. Alexandru, D. E. Quevedo et al.

Cloud computing and distributed computing are becoming ubiquitous in many modern control systems such as smart grids, building automation, robot swarms or intelligent transportation systems. Compared to "isolated" control systems, the advantages of cloud-based and distributed control systems are, in particular, resource pooling and outsourcing, rapid scalability, and high performance. However, these capabilities do not come without risks. In fact, the involved communication and processing of sensitive data via public networks and on third-party platforms promote, among other cyberthreats, eavesdropping and manipulation of data. Encrypted control addresses this security gap and provides confidentiality of the processed data in the entire control loop. This paper presents a tutorial-style introduction to this young but emerging field in the framework of secure control for networked dynamical systems.

Adrian Redder, Arunselvan Ramaswamy, Daniel E. Quevedo

This work considers the problem of control and resource scheduling in networked systems. We present DIRA, a Deep reinforcement learning based Iterative Resource Allocation algorithm, which is scalable and control-aware. Our algorithm is tailored towards large-scale problems where control and scheduling need to act jointly to optimize performance. DIRA can be used to schedule general time-domain optimization based controllers. In the present work, we focus on control designs based on suitably adapted linear quadratic regulators. We apply our algorithm to networked systems with correlated fading communication channels. Our simulations show that DIRA scales well to large scheduling problems.

Burak Demirel, Arunselvan Ramaswamy, Daniel E. Quevedo et al.

We consider networked control systems consisting of multiple independent controlled subsystems, operating over a shared communication network. Such systems are ubiquitous in cyber-physical systems, Internet of Things, and large-scale industrial systems. In many large-scale settings, the size of the communication network is smaller than the size of the system. In consequence, scheduling issues arise. The main contribution of this paper is to develop a deep reinforcement learning-based \emph{control-aware} scheduling (\textsc{DeepCAS}) algorithm to tackle these issues. We use the following (optimal) design strategy: First, we synthesize an optimal controller for each subsystem; next, we design a learning algorithm that adapts to the chosen subsystems (plants) and controllers. As a consequence of this adaptation, our algorithm finds a schedule that minimizes the \emph{control loss}. We present empirical results to show that \textsc{DeepCAS} finds schedules with better performance than periodic ones.

Arunselvan Ramaswamy, Shalabh Bhatnagar, Daniel E. Quevedo

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control problem becomes more tractable, function approximations affect stability and convergence. In this paper, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze Policy Gradient methods and noisy Value Iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between the limiting set and the approximation errors is established. Finally, experimental results are presented that support the theory.

Burak Demirel, Alex S. Leong, Vijay Gupta et al.

This paper studies the optimal output-feedback control of a linear time-invariant system where a stochastic event-based scheduler triggers the communication between the sensor and the controller. The primary goal of the use of this type of scheduling strategy is to provide significant reductions in the usage of the sensor-to-controller communication and, in turn, improve energy expenditure in the network. In this paper, we aim to design an admissible control policy, which is a function of the observed output, to minimize a quadratic cost function while employing a stochastic event-triggered scheduler that preserves the Gaussian property of the plant state and the estimation error. For the infinite horizon case, we present analytical expressions that quantify the trade-off between the communication cost and control performance of such event-triggered control systems. This trade-off is confirmed quantitatively via numerical examples.

Kemi Ding, Subhrakanti Dey, Daniel E. Quevedo et al.

This paper studies remote state estimation under denial-of-service (DoS) attacks. A sensor transmits its local estimate of an underlying physical process to a remote estimator via a wireless communication channel. A DoS attacker is capable to interfere the channel and degrades the remote estimation accuracy. Considering the tactical jamming strategies played by the attacker, the sensor adjusts its transmission power. This interactive process between the sensor and the attacker is studied in the framework of a zero-sum stochastic game. To derive their optimal power schemes, we first discuss the existence of stationary Nash equilibrium (SNE) for this game. We then present the monotone structure of the optimal strategies, which helps reduce the computational complexity of the stochastic game algorithm. Numerical examples are provided to illustrate the obtained results.

Alex S. Leong, Subhrakanti Dey, Daniel E. Quevedo

This paper considers a remote state estimation problem with multiple sensors observing a dynamical process, where sensors transmit local state estimates over an independent and identically distributed (i.i.d.) packet dropping channel to a remote estimator. At every discrete time instant, the remote estimator decides whether each sensor should transmit or not, with each sensor transmission incurring a fixed energy cost. The channel is shared such that collisions will occur if more than one sensor transmits at a time. Performance is quantified via an optimization problem that minimizes a convex combination of the expected estimation error covariance at the remote estimator and expected energy usage across the sensors. For transmission schedules dependent only on the estimation error covariance at the remote estimator, this work establishes structural results on the optimal scheduling which show that 1) for unstable systems, if the error covariance is large then a sensor will always be scheduled to transmit, and 2) there is a threshold-type behaviour in switching from one sensor transmitting to another. Specializing to the single sensor case, these structural results demonstrate that a threshold policy (i.e. transmit if the error covariance exceeds a certain threshold and don't transmit otherwise) is optimal. We also consider the situation where sensors transmit measurements instead of state estimates, and establish structural results including the optimality of threshold policies for the single sensor, scalar case. These results provide a theoretical justification for the use of such threshold policies in variance based event triggered estimation. Numerical studies confirm the qualitative behaviour predicted by our structural results. An extension of the structural results to Markovian packet drops is also outlined.

Mohsen Zamani, Brett Ninness, Daniel Quevedo

In this paper, we study networks of discrete-time linear time-invariant subsystems. Our focus is on situations where subsystems are connected to each other through a time-invariant topology and where there exists a base-station whose aim is to control the subsystems into any desired destinations. However, the base-station can only communicate with some of the subsystems that we refer to as leaders. There are no direct links between the base-station and the rest of subsystems, known as followers, as they are only able to liaise among themselves and with some of the leaders. The current paper formulates this framework as the well-known reachability problem for linear systems. Then to address this problem, we introduce notions of leader-reachability and base-reachability. We present algebraic conditions under which these notions hold. It turns out that if subsystems are represented by minimal state space representations, then base-reachability always holds. Hence, we focus on leader-reachability and investigate the corresponding conditions in detail. We further demonstrate that when the networked system parameters i.e. subsystems' parameters and interconnection matrices, assume generic values then the whole network is both leader-reachable and base-reachable.

Alex S. Leong, Daniel E. Quevedo

This note studies the use of relays to improve the performance of Kalman filtering over packet dropping links. Packet reception probabilities are governed by time-varying fading channel gains, and the sensor and relay transmit powers. We consider situations with multiple sensors and relays, where each relay can either forward one of the sensors' measurements to the gateway/fusion center, or perform a simple linear network coding operation on some of the sensor measurements. Using an expected error covariance performance measure, we consider optimal and suboptimal methods for finding the best relay configuration, and power control problems for optimizing the Kalman filter performance. Our methods show that significant performance gains can be obtained through the use of relays, network coding and power control, with at least 30-40$\%$ less power consumption for a given expected error covariance specification.

Junfeng Wu, Yuzhe Li, Daniel E. Quevedo et al.

We consider sensor transmission power control for state estimation, using a Bayesian inference approach. A sensor node sends its local state estimate to a remote estimator over an unreliable wireless communication channel with random data packet drops. As related to packet dropout rate, transmission power is chosen by the sensor based on the relative importance of the local state estimate. The proposed power controller is proved to preserve Gaussianity of local estimate innovation, which enables us to obtain a closed-form solution of the expected state estimation error covariance. Comparisons with alternative non data-driven controllers demonstrate performance improvement using our approach.