He Bai

Haoran Guo, Haoyang Wang, Zhengxiong Li et al.

Tactile sensing is critical for learning-based dexterous manipulation, yet principled guidelines for sensor placement remain largely absent. While dense sensor arrays provide rich contact feedback, they impose significant hardware costs and can even degrade policy performance by introducing redundant or conflicting inputs. This paper presents the first systematic framework for quantifying the contribution of individual tactile sensors to deep reinforcement learning (DRL) policy performance. We propose a two-stage approach: a coarse empirical pruning phase that reduces the sensor count on the Shadow Hand from 92 to 21 while retaining 93\% task performance, followed by a fine-grained active learning phase that combines Gaussian Process Regression (GPR) with Lasso regression to rank the functional importance of each remaining sensor. Our analysis reveals that sensors on the thumb, ring finger, and little finger dominate manipulation performance, while middle-finger sensors exhibit negative contributions -- actively degrading policy learning. Ablation studies across three manipulation tasks (block, egg, and pen) confirm that a 14-sensor configuration preserves over 90\% of the full-array performance. Zero-shot transfer experiments on two novel objects and cross-platform validation on the Allegro and Leap Hand further demonstrate that the identified importance rankings generalize across tasks and robot morphologies. These findings establish quantitative deployment guidelines that enable practitioners to select cost-effective sensor configurations with predictable performance trade-offs.

Gangshan Jing, He Bai, Jemin George et al.

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

Asma Tabassum, He Bai

This paper develops and evaluates the performance of an allocation agent to be potentially integrated into the onboard Detect and Avoid (DAA) computer of an Unmanned Aircraft System (UAS). We consider a UAS that can be fully controlled by the onboard DAA system and by a remote human pilot. With a communication channel prone to latency, we consider a mixed initiative interaction environment, where the control authority of the UAS is dynamically allocated by the allocation agent. In an encounter with a dynamic intruder, the probability of collision may increase in the absence of pilot commands in the presence of latency. Moreover, a delayed pilot command may not result in safe resolution of the current scenario and need to be improvised. We design an optimization algorithm to reduce collision risk and refine delayed pilot commands. Towards this end, a Markov Decision Process (MDP)and its solution are employed to create a wait time map. The map consists of estimated times that the UAS can wait for the remote pilot commands at each state. A command blending algorithm is designed to select an avoidance maneuver that prioritizes the pilot intention extracted from the pilot commands. The wait time map and the command blending algorithm are implemented and integrated into a closed-loop simulator. We conduct ten thousands fast-time Monte Carlo simulations and compare the performance of the integrated setup with a standalone DAA setup. The simulation results show that the allocation agent enables the UAS to wait without inducing any near mid air collision (NMAC) and severe loss of well clear (LoWC) while positively improve pilot involvement in the encounter resolution.

Ryan Self, Kevin Coleman, He Bai et al.

In this paper, a novel approach to the output-feedback inverse reinforcement learning (IRL) problem is developed by casting the IRL problem, for linear systems with quadratic cost functions, as a state estimation problem. Two observer-based techniques for IRL are developed, including a novel observer method that re-uses previous state estimates via history stacks. Theoretical guarantees for convergence and robustness are established under appropriate excitation conditions. Simulations demonstrate the performance of the developed observers and filters under noisy and noise-free measurements.

Gangshan Jing, He Bai, Jemin George et al.

Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dynamics are not known. To resolve this problem, we propose a parallel RL scheme for a linear quadratic regulator (LQR) design in a continuous-time linear MAS. The idea is to exploit the structural properties of two graphs embedded in the $Q$ and $R$ weighting matrices in the LQR objective to define an orthogonal transformation that can convert the original LQR design to multiple decoupled smaller-sized LQR designs. We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality. Conditions for decomposability, an algorithm for constructing the transformation matrix, a parallel RL algorithm, and robustness analysis when the design is applied to non-homogeneous MAS are presented. Simulations show that the proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost.

Anjaly Parayil, He Bai, Jemin George et al.

Motivated by decentralized approaches to machine learning, we propose a collaborative Bayesian learning algorithm taking the form of decentralized Langevin dynamics in a non-convex setting. Our analysis show that the initial KL-divergence between the Markov Chain and the target posterior distribution is exponentially decreasing while the error contributions to the overall KL-divergence from the additive noise is decreasing in polynomial time. We further show that the polynomial-term experiences speed-up with number of agents and provide sufficient conditions on the time-varying step-sizes to guarantee convergence to the desired distribution. The performance of the proposed algorithm is evaluated on a wide variety of machine learning tasks. The empirical results show that the performance of individual agents with locally available data is on par with the centralized setting with considerable improvement in the convergence rate.

Sayak Mukherjee, He Bai, Aranya Chakrabortty

We present a set of model-free, reduced-dimensional reinforcement learning (RL) based optimal control designs for linear time-invariant singularly perturbed (SP) systems. We first present a state-feedback and output-feedback based RL control design for a generic SP system with unknown state and input matrices. We take advantage of the underlying time-scale separation property of the plant to learn a linear quadratic regulator (LQR) for only its slow dynamics, thereby saving a significant amount of learning time compared to the conventional full-dimensional RL controller. We analyze the sub-optimality of the design using SP approximation theorems and provide sufficient conditions for closed-loop stability. Thereafter, we extend both designs to clustered multi-agent consensus networks, where the SP property reflects through clustering. We develop both centralized and cluster-wise block-decentralized RL controllers for such networks, in reduced dimensions. We demonstrate the details of the implementation of these controllers using simulations of relevant numerical examples and compare them with conventional RL designs to show the computational benefits of our approach.

Sam Allison, He Bai, Balaji Jayaraman

In this article, we study the well known problem of wind estimation in atmospheric turbulence using small unmanned aerial systems (sUAS). We present a machine learning approach to wind velocity estimation based on quadcopter state measurements without a wind sensor. We accomplish this by training a long short-term memory (LSTM) neural network (NN) on roll and pitch angles and quadcopter position inputs with forcing wind velocities as the targets. The datasets are generated using a simulated quadcopter in turbulent wind fields. The trained neural network is deployed to estimate the turbulent winds as generated by the Dryden gust model as well as a realistic large eddy simulation (LES) of a near-neutral atmospheric boundary layer (ABL) over flat terrain. The resulting NN predictions are compared to a wind triangle approach that uses tilt angle as an approximation of airspeed. Results from this study indicate that the LSTM-NN based approach predicts lower errors in both the mean and variance of the local wind field as compared to the wind triangle approach. The work reported in this article demonstrates the potential of machine learning for sensor-less wind estimation and has strong implications to large-scale low-altitude atmospheric sensing using sUAS for environmental and autonomous navigation applications.