Rafael Fierro

Shaifalee Saxena, Rafael Fierro, Alexander Scheinker

Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches.

Shaifalee Saxena, Alan Williams, Rafael Fierro et al.

In this paper, we study the use of robust model independent bounded extremum seeking (ES) feedback control to improve the robustness of deep reinforcement learning (DRL) controllers for a class of nonlinear time-varying systems. DRL has the potential to learn from large datasets to quickly control or optimize the outputs of many-parameter systems, but its performance degrades catastrophically when the system model changes rapidly over time. Bounded ES can handle time-varying systems with unknown control directions, but its convergence speed slows down as the number of tuned parameters increases and, like all local adaptive methods, it can get stuck in local minima. We demonstrate that together, DRL and bounded ES result in a hybrid controller whose performance exceeds the sum of its parts with DRL taking advantage of historical data to learn how to quickly control a many-parameter system to a desired setpoint while bounded ES ensures its robustness to time variations. We present a numerical study of a general time-varying system and a combined ES-DRL controller for automatic tuning of the Low Energy Beam Transport section at the Los Alamos Neutron Science Center linear particle accelerator.

Nicola Bezzo, Joshua P. Hecker, Karl Stolleis et al.

In this paper we consider the problem of coordinating robotic systems with different kinematics, sensing and vision capabilities to achieve certain mission goals. An approach that makes use of a heterogeneous team of agents has several advantages when cost, integration of capabilities, or large search areas need to be considered. A heterogeneous team allows for the robots to become "specialized", accomplish sub-goals more effectively, and thus increase the overall mission efficiency. Two main scenarios are considered in this work. In the first case study we exploit mobility to implement a power control algorithm that increases the Signal to Interference plus Noise Ratio (SINR) among certain members of the network. We create realistic sensing fields and manipulation by using the geometric properties of the sensor field-of-view and the manipulability metric, respectively. The control strategy for each agent of the heterogeneous system is governed by an artificial physics law that considers the different kinematics of the agents and the environment, in a decentralized fashion. Through simulation results we show that the network is able to stay connected at all times and covers the environment well. The second scenario studied in this paper is the biologically-inspired coordination of heterogeneous physical robotic systems. A team of ground rovers, designed to emulate desert seed-harvester ants, explore an experimental area using behaviors fine-tuned in simulation by a genetic algorithm. Our robots coordinate with a base station and collect clusters of resources scattered within the experimental space. We demonstrate experimentally that through coordination with an aerial vehicle, our ant-like ground robots are able to collect resources two times faster than without the use of heterogeneous coordination.

Nicola Bezzo, Patricio J. Cruz Davalos, Francesco Sorrentino et al.

In this paper we propose an application of adaptive synchronization of chaos to detect changes in the topology of a mobile robotic network. We assume that the network may evolve in time due to the relative motion of the mobile robots and due to unknown environmental conditions, such as the presence of obstacles in the environment. We consider that each robotic agent is equipped with a chaotic oscillator whose state is propagated to the other robots through wireless communication, with the goal of synchronizing the oscillators. We introduce an adaptive strategy that each agent independently implements to: (i) estimate the net coupling of all the oscillators in its neighborhood and (ii) synchronize the state of the oscillators onto the same time evolution. We show that by using this strategy, synchronization can be attained and changes in the network topology can be detected. We go one step forward and consider the possibility of using this information to control the mobile network. We show the potential applicability of our technique to the problem of maintaining a formation between a set of mobile platforms, which operate in an inhomogeneous and uncertain environment. We discuss the importance of using chaotic oscillators and validate our methodology by numerical simulations.