Abdullah Al Maruf

Abdullah Al Maruf, Luyao Niu, Bhaskar Ramasubramanian et al.

Autonomous cyber and cyber-physical systems need to perform decision-making, learning, and control in unknown environments. Such decision-making can be sensitive to multiple factors, including modeling errors, changes in costs, and impacts of events in the tails of probability distributions. Although multi-agent reinforcement learning (MARL) provides a framework for learning behaviors through repeated interactions with the environment by minimizing an average cost, it will not be adequate to overcome the above challenges. In this paper, we develop a distributed MARL approach to solve decision-making problems in unknown environments by learning risk-aware actions. We use the conditional value-at-risk (CVaR) to characterize the cost function that is being minimized, and define a Bellman operator to characterize the value function associated to a given state-action pair. We prove that this operator satisfies a contraction property, and that it converges to the optimal value function. We then propose a distributed MARL algorithm called the CVaR QD-Learning algorithm, and establish that value functions of individual agents reaches consensus. We identify several challenges that arise in the implementation of the CVaR QD-Learning algorithm, and present solutions to overcome these. We evaluate the CVaR QD-Learning algorithm through simulations, and demonstrate the effect of a risk parameter on value functions at consensus.

Arezoo Rajabi, Bhaskar Ramasubramanian, Abdullah Al Maruf et al.

Cyber and cyber-physical systems equipped with machine learning algorithms such as autonomous cars share environments with humans. In such a setting, it is important to align system (or agent) behaviors with the preferences of one or more human users. We consider the case when an agent has to learn behaviors in an unknown environment. Our goal is to capture two defining characteristics of humans: i) a tendency to assess and quantify risk, and ii) a desire to keep decision making hidden from external parties. We incorporate cumulative prospect theory (CPT) into the objective of a reinforcement learning (RL) problem for the former. For the latter, we use differential privacy. We design an algorithm to enable an RL agent to learn policies to maximize a CPT-based objective in a privacy-preserving manner and establish guarantees on the privacy of value functions learned by the algorithm when rewards are sufficiently close. This is accomplished through adding a calibrated noise using a Gaussian process mechanism at each step. Through empirical evaluations, we highlight a privacy-utility tradeoff and demonstrate that the RL agent is able to learn behaviors that are aligned with that of a human user in the same environment in a privacy-preserving manner

Eman Badr, Abdullah Al Maruf

Complex, interconnected cyber-physical systems (CPS) are increasingly prevalent in domains such as power systems. Cyber-resilient architectures have been proposed to recover compromised cyber components of CPS. Recent works have studied tuning the recovery times of such architectures to guarantee safety in single-system settings. Extending these designs to interconnected CPS is more challenging, since solutions must account for attacks on multiple subsystems that can occur in any order and potentially infinite possible temporal overlap. This paper aims to address the aforementioned challenge by developing a scalable framework to assign resilient architectures and to inform the tuning of their recovery times. Our approach introduces a scalar index that quantifies the impact of each subsystem on safety under compromised input. These indices aggregate linearly across subsystems, enabling scalable analysis under arbitrary attack orderings and temporal overlaps. We establish a linear inequality relating each subsystem's index and recovery time that guarantees safety and guides resilient architecture assignment. We also propose a segmentation-based approach to strengthen the previously derived conditions. We then present algorithms to compute the proposed indices and to find a cost-optimal architecture assignment with a safety guarantee. We validate the framework through a case study on temperature regulation in interconnected rooms under different attack scenarios.