Hamidreza Modares

Amir Modares, Zhen Zhang, Themistoklis Charalambous et al.

This paper develops a data-driven reachability framework for linear systems whose disturbances are modeled by probabilistic zonotopes (PZs), combining bounded deterministic and Gaussian stochastic components. In contrast to methods that require a precisely known disturbance model (either purely deterministic or purely stochastic), we assume only a conservative prior PZ and refine it from data. The framework separates two uncertainty sources: realized disturbances, which act along the collected trajectory and govern the size of the data-consistent model set, and aleatory disturbances, which enter as future additive uncertainty during reachable-set propagation; both shape the reachable sets, but through different mechanisms. Refinement exploits prior system knowledge together with trajectory-consistency constraints induced by the data, which impose affine couplings between deterministic and Gaussian latent variables. We accordingly develop a constrained-PZ calculus that absorbs the stochastic part of these constraints into an equivalent representation, removes infeasible latent directions, and reduces stochastic covariance, together with identification-aware fusion rules for combining heterogeneous constrained-PZ descriptions. The refined realized-disturbance proxies then serve as scenarios in a linear program that learns the smallest translated and scaled copy of the prior disturbance set that contains all proxy confidence sets while remaining nested in the prior. The resulting deterministic, high-probability reachable sets carry formal containment guarantees with substantially reduced conservatism, and numerical examples confirm that the pipeline tightens both the data-consistent model set and the propagated reachable sets.

Babak Esmaeili, Nariman Niknejad, Hamidreza Modares

This paper presents a risk-aware safe reinforcement learning (RL) control design for stochastic discrete-time linear systems. Rather than using a safety certifier to myopically intervene with the RL controller, a risk-informed safe controller is also learned besides the RL controller, and the RL and safe controllers are combined together. Several advantages come along with this approach: 1) High-confidence safety can be certified without relying on a high-fidelity system model and using limited data available, 2) Myopic interventions and convergence to an undesired equilibrium can be avoided by deciding on the contribution of two stabilizing controllers, and 3) highly efficient and computationally tractable solutions can be provided by optimizing over a scalar decision variable and linear programming polyhedral sets. To learn safe controllers with a large invariant set, piecewise affine controllers are learned instead of linear controllers. To this end, the closed-loop system is first represented using collected data, a decision variable, and noise. The effect of the decision variable on the variance of the safe violation of the closed-loop system is formalized. The decision variable is then designed such that the probability of safety violation for the learned closed-loop system is minimized. It is shown that this control-oriented approach reduces the data requirements and can also reduce the variance of safety violations. Finally, to integrate the safe and RL controllers, a new data-driven interpolation technique is introduced. This method aims to maintain the RL agent's optimal implementation while ensuring its safety within environments characterized by noise. The study concludes with a simulation example that serves to validate the theoretical results.

Aquib Mustafa, Majid Mazouchi, Subramanya Nageshrao et al.

Reinforcement learning (RL) agents with pre-specified reward functions cannot provide guaranteed safety across variety of circumstances that an uncertain system might encounter. To guarantee performance while assuring satisfaction of safety constraints across variety of circumstances, an assured autonomous control framework is presented in this paper by empowering RL algorithms with metacognitive learning capabilities. More specifically, adapting the reward function parameters of the RL agent is performed in a metacognitive decision-making layer to assure the feasibility of RL agent. That is, to assure that the learned policy by the RL agent satisfies safety constraints specified by signal temporal logic while achieving as much performance as possible. The metacognitive layer monitors any possible future safety violation under the actions of the RL agent and employs a higher-layer Bayesian RL algorithm to proactively adapt the reward function for the lower-layer RL agent. To minimize the higher-layer Bayesian RL intervention, a fitness function is leveraged by the metacognitive layer as a metric to evaluate success of the lower-layer RL agent in satisfaction of safety and liveness specifications, and the higher-layer Bayesian RL intervenes only if there is a risk of lower-layer RL failure. Finally, a simulation example is provided to validate the effectiveness of the proposed approach.