Giovanni Russo

Marco Di Francesco, Simone Fagioli, Massimiliano D. Rosini et al.

We review recent results and present new ones on a deterministic follow-the-leader particle approximation of first and second order models for traffic flow and pedestrian movements. We start by constructing the particle scheme for the first order Lighthill-Whitham-Richards (LWR) model for traffic flow. The approximation is performed by a set of ODEs following the position of discretised vehicles seen as moving particles. The convergence of the scheme in the many particle limit towards the unique entropy solution of the LWR equation is proven in the case of the Cauchy problem on the real line. We then extend our approach to the Initial-Boundary Value Problem (IBVP) with time-varying Dirichlet data on a bounded interval. In this case we prove that our scheme is convergent strongly in $L^1$ up to a subsequence. We then review extensions of this approach to the Hughes model for pedestrian movements and to the second order Aw-Rascle-Zhang (ARZ) model for vehicular traffic. Finally, we complement our results with numerical simulations. In particular, the simulations performed on the IBVP and the ARZ model suggest the consistency of the corresponding schemes, which is easy to prove rigorously in some simple cases.

Yingqi Gu, Mingming Liu, Joe Naoum-Sawaya et al.

Electric Vehicles (EVs) and Plug-in Hybrid Electric Vehicles (PHEVs) are increasingly being seen as a means of mitigating the pressing concerns of traffic-related pollution. While hybrid vehicles are usually designed with the objective of minimising fuel consumption, in this paper we propose engine management strategies that also take into account environmental effects of the vehicles to pedestrians outside of the vehicles. Specifically, we present optimisation based engine energy management strategies for PHEVs, that attempt to minimise the environmental impact of pedestrians along the route of the vehicle, while taking account of route dependent uncertainties. We implement the proposed approach in a real PHEV, and evaluate the performance in a hardware-in-the-loop platform. A variety of simulation results are given to illustrate the efficacy of our proposed approach.

Emiland Garrabe, Hozefa Jesawada, Carmen Del Vecchio et al.

This paper is concerned with a finite-horizon inverse control problem, which has the goal of reconstructing, from observations, the possibly non-convex and non-stationary cost driving the actions of an agent. In this context, we present a result enabling cost reconstruction by solving an optimization problem that is convex even when the agent cost is not and when the underlying dynamics is nonlinear, non-stationary and stochastic. To obtain this result, we also study a finite-horizon forward control problem that has randomized policies as decision variables. We turn our findings into algorithmic procedures and show the effectiveness of our approach via in-silico and hardware validations. All experiments confirm the effectiveness of our approach.

Shaun Sweeney, Rodrigo Ordonez-Hurtado, Francesco Pilla et al.

The effect of transport-related pollution on human health is fast becoming recognised as a major issue in cities worldwide. Cyclists, in particular, face great risks, as they typically are most exposed to tail-pipe emissions. Three avenues are being explored worldwide in the fight against urban pollution: (i) outright bans on polluting vehicles and embracing zero tailpipe emission vehicles; (ii) measuring air-quality as a means to better informing citizens of zones of higher pollution; and (iii) developing smart mobility devices that seek to minimize the effect of polluting devices on citizens as they transport goods and individuals in our cities. Following this latter direction, in this paper we present a new way to protect cyclists from the effect of urban pollution. Namely, by exploiting the actuation possibilities afforded by pedelecs or e-bikes (electric bikes), we design a cyber-physical system that mitigates the effect of urban pollution by indirectly controlling the breathing rate of cyclists in polluted areas. Results from a real device are presented to illustrate the efficacy of our system.

Francesco De Lellis, Marco Coraggio, Giovanni Russo et al.

One of the major challenges in Deep Reinforcement Learning for control is the need for extensive training to learn the policy. Motivated by this, we present the design of the Control-Tutored Deep Q-Networks (CT-DQN) algorithm, a Deep Reinforcement Learning algorithm that leverages a control tutor, i.e., an exogenous control law, to reduce learning time. The tutor can be designed using an approximate model of the system, without any assumption about the knowledge of the system's dynamics. There is no expectation that it will be able to achieve the control objective if used stand-alone. During learning, the tutor occasionally suggests an action, thus partially guiding exploration. We validate our approach on three scenarios from OpenAI Gym: the inverted pendulum, lunar lander, and car racing. We demonstrate that CT-DQN is able to achieve better or equivalent data efficiency with respect to the classic function approximation solutions.

Antonio Baeza, Sebastiano Boscarino, Pep Mulet et al.

In [Baeza et al., Computers and Fluids, 159, 156--166 (2017)] a new method for the numerical solution of ODEs is presented. This methods can be regarded as an approximate formulation of the Taylor methods and it follows an approach that has a much easier implementation than the original Taylor methods, since only the functions in the ODEs, and not their high order derivatives, are needed. In this reference, the absolute stability region of the new methods is conjectured to be coincident with that of their exact counterparts. There is also a conjecture about their relationship with Runge-Kutta methods. In this work we answer positively both conjectures.

Veronica Centorrino, Francesca Rossi, Francesco Bullo et al.

This paper studies equality-constrained composite minimization problems. This class of problems, capturing regularization terms and inequality constraints, naturally arises in a wide range of engineering and machine learning applications. To tackle these optimization problems, inspired by recent results, we introduce the \emph{proportional--integral proximal gradient dynamics} (PI--PGD): a closed-loop system where the Lagrange multipliers are control inputs and states are the problem decision variables. First, we establish the equivalence between the stationary points of the minimization problem and the equilibria of the PI--PGD. Then for the case of affine constraints, by leveraging tools from contraction theory we give a comprehensive convergence analysis for the dynamics, showing linear--exponential convergence towards the equilibrium. That is, the distance between each solution and the equilibrium is upper bounded by a function that first decreases linearly and then exponentially. Our findings are illustrated numerically on a set of representative examples, which include an exploratory application to nonlinear equality constraints.

Sara Maria Brancato, Francesco De Lellis, Davide Salzano et al.

We investigate the problem of using a learning-based strategy to stabilize a synthetic toggle switch via an external control approach. To overcome the data efficiency problem that would render the algorithm unfeasible for practical use in synthetic biology, we adopt a sim-to-real paradigm where the policy is learnt via training on a simplified model of the toggle switch and it is then subsequently exploited to control a more realistic model of the switch parameterized from in-vivo experiments. Our in-silico experiments confirm the viability of the approach suggesting its potential use for in-vivo control implementations.

Francesco De Lellis, Marco Coraggio, Giovanni Russo et al.

In addressing control problems such as regulation and tracking through reinforcement learning, it is often required to guarantee that the acquired policy meets essential performance and stability criteria such as a desired settling time and steady-state error prior to deployment. Motivated by this necessity, we present a set of results and a systematic reward shaping procedure that (i) ensures the optimal policy generates trajectories that align with specified control requirements and (ii) allows to assess whether any given policy satisfies them. We validate our approach through comprehensive numerical experiments conducted in two representative environments from OpenAI Gym: the Inverted Pendulum swing-up problem and the Lunar Lander. Utilizing both tabular and deep reinforcement learning methods, our experiments consistently affirm the efficacy of our proposed framework, highlighting its effectiveness in ensuring policy adherence to the prescribed control requirements.

Julien Monteil, Giovanni Russo, Robert Shorten

This paper is concerned with the study of bidirectionally coupled platoon systems. The case considered is when the vehicles are heterogeneous and the coupling can be nonlinear and asymmetric. For such systems, a sufficient condition for $\mathcal{L}_{\infty}$ string stability is presented. The effectiveness of our approach is illustrated via a numerical example, where it is shown how our result can be recast as an optimization problem, allowing to design the control protocol for each vehicle independently on the other vehicles and hence leading to a bottom-up approach for the design of string stable systems able to track a time-varying reference speed.

Rodrigo Ordóñez-Hurtado, Giovanni Russo, Sam Sinnott et al.

Vehicles are becoming connected entities. As a result, a likely scenario is that such entities might be literally bombarded with information from a multitude of devices. In this context, a key challenging requirement for both connected and autonomous vehicles is that they will need to become cognitive bodies, able to parse information and use only the pieces of information that are relevant to the vehicle in the context of a given journey. In order to address this fundamental requirement, a decision engine is presented in this paper. The engine makes it possible for the vehicle to understand which pieces of information are really relevant, and subsequently to process only those pieces of information. In order to illustrate the key features of our system, we show that it is possible to build upon the engine to develop a distributed traffic management system, and then we validate such a system via both conventional (numerical and SUMO-based) simulations and a Hardware-in-the-Loop (HIL) platform. Both the conventional simulations and the HIL validation showed that the engine can be effectively used to design a distributed traffic management system.

Giovanni Russo, Rovert Shorten

This paper is concerned with the study of synchronization and consensus phenomena in complex networks of diffusively-coupled nodes subject to external disturbances. Specifically, we make use of stochastic Lyapunov functions to provide conditions for synchronization and consensus for networks of nonlinear, diffusively coupled nodes, where noise diffusion is not just additive but it depends on the nodes' state. The sufficient condition we provide, wich links together network topology, coupling strength and noise diffusion, offers two interesting interpretations. First, as suggested by {\em intuition}, in order for a network to achieve synchronization/consensus, its nodes need to be sufficiently well connected together. The second implication might seem, instead, counter-intuitive: if noise diffusion is {\em properly} designed, then it can drive an unsynchronized network towards synchronization/consensus. Motivated by our current research in Smart Cities and Internet of Things, we illustrate the effectiveness of our approach by showing how our results can be used to control certain collective decision processes.

Sara Maria Brancato, Davide Salzano, Francesco De Lellis et al.

A key problem toward the use of microorganisms as bio-factories is reaching and maintaining cellular communities at a desired density and composition so that they can efficiently convert their biomass into useful compounds. Promising technological platforms for the real time, scalable control of cellular density are bioreactors. In this work, we developed a learning-based strategy to expand the toolbox of available control algorithms capable of regulating the density of a \textit{single} bacterial population in bioreactors. Specifically, we used a sim-to-real paradigm, where a simple mathematical model, calibrated using a few data, was adopted to generate synthetic data for the training of the controller. The resulting policy was then exhaustively tested in vivo using a low-cost bioreactor known as Chi.Bio, assessing performance and robustness. In addition, we compared the performance with more traditional controllers (namely, a PI and an MPC), confirming that the learning-based controller exhibits similar performance in vivo. Our work showcases the viability of learning-based strategies for the control of cellular density in bioreactors, making a step forward toward their use for the control of the composition of microbial consortia.

Hozefa Jesawada, Antonio Acernese, Giovanni Russo et al.

Ensuring robustness against epistemic, possibly adversarial, perturbations is essential for reliable real-world decision-making. While the Probabilistic Ensembles with Trajectory Sampling (PETS) algorithm inherently handles uncertainty via ensemble-based probabilistic models, it lacks guarantees against structured adversarial or worst-case uncertainty distributions. To address this, we propose DR-PETS, a distributionally robust extension of PETS that certifies robustness against adversarial perturbations. We formalize uncertainty via a p-Wasserstein ambiguity set, enabling worst-case-aware planning through a min-max optimization framework. While PETS passively accounts for stochasticity, DR-PETS actively optimizes robustness via a tractable convex approximation integrated into PETS planning loop. Experiments on pendulum stabilization and cart-pole balancing show that DR-PETS certifies robustness against adversarial parameter perturbations, achieving consistent performance in worst-case scenarios where PETS deteriorates.

Enrico Ferrentino, Pasquale Chiacchio, Giovanni Russo

We present the principled design of a control pipeline for the synthesis of policies from examples data. The pipeline, based on a discretized design which we term as discrete fully probabilistic design, expounds an algorithm recently introduced in Gagliardi and Russo (2021) to synthesize policies from examples for constrained, stochastic and nonlinear systems. Contrary to other approaches, the pipeline we present: (i) does not need the constraints to be fulfilled in the possibly noisy example data; (ii) enables control synthesis even when the data are collected from an example system that is different from the one under control. The design is benchmarked numerically on an example that involves controlling an inverted pendulum with actuation constraints starting from data collected from a physically different pendulum that does not satisfy the system-specific actuation constraints. We also make our fully documented code openly available.

F. De Lellis, M. Coraggio, G. Russo et al.

We present an architecture where a feedback controller derived on an approximate model of the environment assists the learning process to enhance its data efficiency. This architecture, which we term as Control-Tutored Q-learning (CTQL), is presented in two alternative flavours. The former is based on defining the reward function so that a Boolean condition can be used to determine when the control tutor policy is adopted, while the latter, termed as probabilistic CTQL (pCTQL), is instead based on executing calls to the tutor with a certain probability during learning. Both approaches are validated, and thoroughly benchmarked against Q-Learning, by considering the stabilization of an inverted pendulum as defined in OpenAI Gym as a representative problem.

Francesco De Lellis, Giovanni Russo, Mario di Bernardo

We introduce a control-tutored reinforcement learning (CTRL) algorithm. The idea is to enhance tabular learning algorithms by means of a control strategy with limited knowledge of the system model. By tutoring the learning process, the learning rate can be substantially reduced. We use the classical problem of stabilizing an inverted pendulum as a benchmark to numerically illustrate the advantages and disadvantages of the approach.

Francesco De Lellis, Fabrizia Auletta, Giovanni Russo et al.

We introduce a control-tutored reinforcement learning (CTRL) algorithm. The idea is to enhance tabular learning algorithms so as to improve the exploration of the state-space, and substantially reduce learning times by leveraging some limited knowledge of the plant encoded into a tutoring model-based control strategy. We illustrate the benefits of our novel approach and its effectiveness by using the problem of controlling one or more agents to herd and contain within a goal region a set of target free-roving agents in the plane.

Francesco De Lellis, Fabrizia Auletta, Giovanni Russo et al.

In this extended abstract we introduce a novel control-tutored Q-learning approach (CTQL) as part of the ongoing effort in developing model-based and safe RL for continuous state spaces. We validate our approach by applying it to a challenging multi-agent herding control problem.

Meghana Rathi, Pietro Ferraro, Giovanni Russo

In this paper we propose a new approach to complement reinforcement learning (RL) with model-based control (in particular, Model Predictive Control - MPC). We introduce an algorithm, the MPC augmented RL (MPRL) that combines RL and MPC in a novel way so that they can augment each other's strengths. We demonstrate the effectiveness of the MPRL by letting it play against the Atari game Pong. For this task, the results highlight how MPRL is able to outperform both RL and MPC when these are used individually.