Yongcan Cao

Yongcan Cao, Wei Ren

This paper focuses on analyzing the finite-time convergence of a nonlinear consensus algorithm for multi-agent networks with unknown inherent nonlinear dynamics. Due to the existence of the unknown inherent nonlinear dynamics, the stability analysis and the finite-time convergence analysis of the closed-loop system under the proposed consensus algorithm are more challenging than those under the well-studied consensus algorithms for known linear systems. For this purpose, we propose a novel stability tool based on a generalized comparison lemma. With the aid of the novel stability tool, it is shown that the proposed nonlinear consensus algorithm can guarantee finite-time convergence if the directed switching interaction graph has a directed spanning tree at each time interval. Specifically, the finite-time convergence is shown by comparing the closed-loop system under the proposed consensus algorithm with some well-designed closed-loop system whose stability properties are easier to obtain. Moreover, the stability and the finite-time convergence of the closed-loop system using the proposed consensus algorithm under a (general) directed switching interaction graph can even be guaranteed by the stability and the finite-time convergence of some special well-designed nonlinear closed-loop system under some special directed switching interaction graph, where each agent has at most one neighbor whose state is either the maximum of those states that are smaller than its own state or the minimum of those states that are larger than its own state. This provides a stimulating example for the potential applications of the proposed novel stability tool in the stability analysis of linear/nonlinear closed-loop systems by making use of known results in linear/nonlinear systems. For illustration of the theoretical result, we provide a simulation example.

Devin White, Mingkang Wu, Ellen Novoseller et al.

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforcement learning paradigms, based on human relative preferences over sample pairs, the proposed rating-based reinforcement learning approach is based on human evaluation of individual trajectories without relative comparisons between sample pairs. The rating-based reinforcement learning approach builds on a new prediction model for human ratings and a novel multi-class loss function. We conduct several experimental studies based on synthetic ratings and real human ratings to evaluate the effectiveness and benefits of the new rating-based reinforcement learning approach.

Umer Siddique, Abhinav Sinha, Yongcan Cao

In this paper, we address the issue of fairness in preference-based reinforcement learning (PbRL) in the presence of multiple objectives. The main objective is to design control policies that can optimize multiple objectives while treating each objective fairly. Toward this objective, we design a new fairness-induced preference-based reinforcement learning or FPbRL. The main idea of FPbRL is to learn vector reward functions associated with multiple objectives via new welfare-based preferences rather than reward-based preference in PbRL, coupled with policy learning via maximizing a generalized Gini welfare function. Finally, we provide experiment studies on three different environments to show that the proposed FPbRL approach can achieve both efficiency and equity for learning effective and fair policies.

David W. Casbeer, Yongcan Cao, Eloy Garcia et al.

In this paper, the problem of bridge consensus is presented and solved. Bridge consensus consists of a network of nodes, some of whom are participating and others are non-participating. The objective is for all the agents to reach average consensus of the participating nodes initial values in a distributed and scalable manner. To do this, the nodes must use the network connections of the non-participating nodes, which act as bridges for information and ignore the initial values of the non-participating nodes. The solution to this problem is made by merging the ideas from estimation theory and consensus theory. By considering the participating nodes has having equal information and the non-participating nodes as having no information, the nodes initial values are transformed into information space. Two consensus filters are run in parallel on the information state and information matrix. Conditions ensuring that the product of the inverse information matrix and the information state of each agent reaches average consensus of the participating agents' initial values is given.

Umer Siddique, Abhinav Sinha, Yongcan Cao

In this paper, we propose an adaptive event-triggered reinforcement learning control for continuous-time nonlinear systems, subject to bounded uncertainties, characterized by complex interactions. Specifically, the proposed method is capable of jointly learning both the control policy and the communication policy, thereby reducing the number of parameters and computational overhead when learning them separately or only one of them. By augmenting the state space with accrued rewards that represent the performance over the entire trajectory, we show that accurate and efficient determination of triggering conditions is possible without the need for explicit learning triggering conditions, thereby leading to an adaptive non-stationary policy. Finally, we provide several numerical examples to demonstrate the effectiveness of the proposed approach.

Peilang Li, Umer Siddique, Yongcan Cao

Deep reinforcement learning (RL) has shown remarkable success in complex domains, however, the inherent black box nature of deep neural network policies raises significant challenges in understanding and trusting the decision-making processes. While existing explainable RL methods provide local insights, they fail to deliver a global understanding of the model, particularly in high-stakes applications. To overcome this limitation, we propose a novel model-agnostic approach that bridges the gap between explainability and interpretability by leveraging Shapley values to transform complex deep RL policies into transparent representations. The proposed approach offers two key contributions: a novel approach employing Shapley values to policy interpretation beyond local explanations and a general framework applicable to off-policy and on-policy algorithms. We evaluate our approach with three existing deep RL algorithms and validate its performance in two classic control environments. The results demonstrate that our approach not only preserves the original models' performance but also generates more stable interpretable policies.

Evelyn Rose, Devin White, Mingkang Wu et al.

This paper explores multiple optimization methods to improve the performance of rating-based reinforcement learning (RbRL). RbRL, a method based on the idea of human ratings, has been developed to infer reward functions in reward-free environments for the subsequent policy learning via standard reinforcement learning, which requires the availability of reward functions. Specifically, RbRL minimizes the cross entropy loss that quantifies the differences between human ratings and estimated ratings derived from the inferred reward. Hence, a low loss means a high degree of consistency between human ratings and estimated ratings. Despite its simple form, RbRL has various hyperparameters and can be sensitive to various factors. Therefore, it is critical to provide comprehensive experiments to understand the impact of various hyperparameters on the performance of RbRL. This paper is a work in progress, providing users some general guidelines on how to select hyperparameters in RbRL.

Conor Wallace, Umer Siddique, Yongcan Cao

Agent modeling is a critical component in developing effective policies within multi-agent systems, as it enables agents to form beliefs about the behaviors, intentions, and competencies of others. Many existing approaches assume access to other agents' episodic trajectories, a condition often unrealistic in real-world applications. Consequently, a practical agent modeling approach must learn a robust representation of the policies of the other agents based only on the local trajectory of the controlled agent. In this paper, we propose \texttt{TransAM}, a novel transformer-based agent modeling approach to encode local trajectories into an embedding space that effectively captures the policies of other agents. We evaluate the performance of the proposed method in cooperative, competitive, and mixed multi-agent environments. Extensive experimental results demonstrate that our approach generates strong policy representations, improves agent modeling, and leads to higher episodic returns.

Mingkang Wu, Devin White, Evelyn Rose et al.

Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.

Huixin Zhan, Feng Tao, Yongcan Cao

Human demonstrations can provide trustful samples to train reinforcement learning algorithms for robots to learn complex behaviors in real-world environments. However, obtaining sufficient demonstrations may be impractical because many behaviors are difficult for humans to demonstrate. A more practical approach is to replace human demonstrations by human queries, i.e., preference-based reinforcement learning. One key limitation of the existing algorithms is the need for a significant amount of human queries because a large number of labeled data is needed to train neural networks for the approximation of a continuous, high-dimensional reward function. To reduce and minimize the need for human queries, we propose a new GAN-assisted human preference-based reinforcement learning approach that uses a generative adversarial network (GAN) to actively learn human preferences and then replace the role of human in assigning preferences. The adversarial neural network is simple and only has a binary output, hence requiring much less human queries to train. Moreover, a maximum entropy based reinforcement learning algorithm is designed to shape the loss towards the desired regions or away from the undesired regions. To show the effectiveness of the proposed approach, we present some studies on complex robotic tasks without access to the environment reward in a typical MuJoCo robot locomotion environment. The obtained results show our method can achieve a reduction of about 99.8% human time without performance sacrifice.

Feng Tao, Rengan Suresh, Johnathan Votion et al.

In this paper, we focus on developing a novel unsupervised machine learning algorithm, named graph based multi-layer k-means++ (G-MLKM), to solve data-target association problem when targets move on a constrained space and minimal information of the targets can be obtained by sensors. Instead of employing the traditional data-target association methods that are based on statistical probabilities, the G-MLKM solves the problem via data clustering. We first will develop the Multi-layer K-means++ (MLKM) method for data-target association at local space given a simplified constrained space situation. Then a p-dual graph is proposed to represent the general constrained space when local spaces are interconnected. Based on the dual graph and graph theory, we then generalize MLKM to G-MLKM by first understanding local data-target association and then extracting cross-local data-target association mathematically analyze the data association at intersections of that space. To exclude potential data-target association errors that disobey physical rules, we also develop error correction mechanisms to further improve the accuracy. Numerous simulation examples are conducted to demonstrate the performance of G-MLKM.

Feng Tao, Yongcan Cao

In this paper, we study the problem of obtaining a control policy that can mimic and then outperform expert demonstrations in Markov decision processes where the reward function is unknown to the learning agent. One main relevant approach is the inverse reinforcement learning (IRL), which mainly focuses on inferring a reward function from expert demonstrations. The obtained control policy by IRL and the associated algorithms, however, can hardly outperform expert demonstrations. To overcome this limitation, we propose a novel method that enables the learning agent to outperform the demonstrator via a new concurrent reward and action policy learning approach. In particular, we first propose a new stereo utility definition that aims to address the bias in the interpretation of expert demonstrations. We then propose a loss function for the learning agent to learn reward and action policies concurrently such that the learning agent can outperform expert demonstrations. The performance of the proposed method is first demonstrated in OpenAI environments. Further efforts are conducted to experimentally validate the proposed method via an indoor drone flight scenario.

Huixin Zhan, Wei-Ming Lin, Yongcan Cao

Besides accuracy, the model size of convolutional neural networks (CNN) models is another important factor considering limited hardware resources in practical applications. For example, employing deep neural networks on mobile systems requires the design of accurate yet fast CNN for low latency in classification and object detection. To fulfill the need, we aim at obtaining CNN models with both high testing accuracy and small size to address resource constraints in many embedded devices. In particular, this paper focuses on proposing a generic reinforcement learning-based model compression approach in a two-stage compression pipeline: pruning and quantization. The first stage of compression, i.e., pruning, is achieved via exploiting deep reinforcement learning (DRL) to co-learn the accuracy and the FLOPs updated after layer-wise channel pruning and element-wise variational pruning via information dropout. The second stage, i.e., quantization, is achieved via a similar DRL approach but focuses on obtaining the optimal bits representation for individual layers. We further conduct experimental results on CIFAR-10 and ImageNet datasets. For the CIFAR-10 dataset, the proposed method can reduce the size of VGGNet by 9x from 20.04MB to 2.2MB with a slight accuracy increase. For the ImageNet dataset, the proposed method can reduce the size of VGG-16 by 33x from 138MB to 4.14MB with no accuracy loss.

Huixin Zhan, Yongcan Cao

Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple, yet often conflicting objectives. A typical approach to obtain optimal policies is to first construct a loss function that is based on the scalarization of individual objectives, and then find the optimal policy that minimizes the loss. However, optimizing the scalarized (and weighted) loss does not necessarily provide a guarantee of high performance on each possibly conflicting objective. In this paper, we propose a vector value based reinforcement learning approach that seeks to explicitly learn the inter-objective relationship and optimize multiple objectives based on the learned relationship. In particular, the proposed method is to first define relationship matrix, a mathematical representation of the inter-objective relationship, and then create one actor and multiple critics that can co-learn the relationship matrix and action selection. The proposed approach can quantify the inter-objective relationship via reinforcement learning when the impact of one objective on another is unknown a prior. We also provide rigorous convergence analysis of the proposed approach and present a quantitative evaluation of the approach based on two testing scenarios.

Huixin Zhan, Yongcan Cao

Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple, yet often conflicting objectives. A typical approach to obtain optimal policies is to first construct a loss function that is based on the scalarization of individual objectives, and then find the optimal policy that minimizes the loss. However, optimizing the scalarized (and weighted) loss does not necessarily provide guarantee of high performance on each possibly conflicting objective because it is challenging to assign the right weights without knowing the relationship among these objectives. Moreover, the effectiveness of these gradient descent algorithms is limited by the agent's ability to explain their decisions and actions to human users. The purpose of this study is two-fold. First, we propose a vector value function based multi-objective reinforcement learning (V2f-MORL) approach that seeks to quantify the inter-objective relationship via reinforcement learning (RL) when the impact of one objective on others is unknown a prior. In particular, we construct one actor and multiple critics that can co-learn the policy and inter-objective relationship matrix (IORM), quantifying the impact of objectives on each other, in an iterative way. Second, we provide a semantic representation that can uncover the trade-off of decision policies made by users to reconcile conflicting objectives based on the proposed V2f-MORL approach for the explainability of the generated behaviors subject to given optimization objectives. We demonstrate the effectiveness of the proposed approach via a MuJoCo based robotics case study.

Samuel Silva, Rengan Suresh, Feng Tao et al.

Data-target association is an important step in multi-target localization for the intelligent operation of un- manned systems in numerous applications such as search and rescue, traffic management and surveillance. The objective of this paper is to present an innovative data association learning approach named multi-layer K-means (MLKM) based on leveraging the advantages of some existing machine learning approaches, including K-means, K-means++, and deep neural networks. To enable the accurate data association from different sensors for efficient target localization, MLKM relies on the clustering capabilities of K-means++ structured in a multi-layer framework with the error correction feature that is motivated by the backpropogation that is well-known in deep learning research. To show the effectiveness of the MLKM method, numerous simulation examples are conducted to compare its performance with K-means, K-means++, and deep neural networks.

Johnathan Votion, Yongcan Cao

This paper focuses on developing new navigation and reconnaissance capabilities for cooperative unmanned systems in uncertain environments. The goal is to design a cooperative multi-vehicle system that can survey an unknown environment and find the most valuable route for personnel to travel. To accomplish the goal, the multi-vehicle system first explores spatially diverse routes and then selects the safest route. In particular, the proposed cooperative path planner sequentially generates a set of spatially diverse routes according to a number of factors, including travel distance, ease of travel, and uncertainty associated with the ease of travel. The planner's dependence on each of these factors is altered by a weighted score, doing so changes the criteria for determining an optimum route. To penalize the selection of same paths by different vehicles, a control gain is used to increase the cost of paths that lie near the route(s) assigned to other vehicles. By varying the control gain, the spatial diversity among routes can be accomplished. By repeatedly searching for different paths cooperatively, an optimal path can be selected that yields the most valuable route.

Kasthurirengan Suresh, Samuel Silva, Johnathan Votion et al.

Data-target pairing is an important step towards multi-target localization for the intelligent operation of unmanned systems. Target localization plays a crucial role in numerous applications, such as search, and rescue missions, traffic management and surveillance. The objective of this paper is to present an innovative target location learning approach, where numerous machine learning approaches, including K-means clustering and supported vector machines (SVM), are used to learn the data pattern across a list of spatially distributed sensors. To enable the accurate data association from different sensors for accurate target localization, appropriate data pre-processing is essential, which is then followed by the application of different machine learning algorithms to appropriately group data from different sensors for the accurate localization of multiple targets. Through simulation examples, the performance of these machine learning algorithms is quantified and compared.

Yongcan Cao, Eduardo Pasiliao

In this paper, we study the distributed control of networked cyber-physical systems when a much more energy-efficient distributed communication management strategy is proposed to solve the well-studied consensus problem. In contrast to the existing potential-based network topology control method, the proposed topology control method is based on the variation of communication ranges such that each agent can control its ad hoc communication range. The proposed network topology control technique can not only guarantee network connectivity but also reduce the communication energy. We apply the new network topology control technique, based on variable communication ranges, in a well-studied consensus problem, where the communication range for each agent is designed locally along with a new bounded control algorithm. Theoretical analysis is then provided to show that the proposed network topology control technique can guarantee consensus with bounded communication energy consumption. Finally, simulation examples are provided to show the effectiveness of the proposed energy-efficient distributed topology control technique.

Mohsen Khalili, Xiaodong Zhang, Marios M. Polycarpou et al.

This paper presents an adaptive fault-tolerant control (FTC) scheme for a class of nonlinear uncertain multi-agent systems. A local FTC scheme is designed for each agent using local measurements and suitable information exchanged between neighboring agents. Each local FTC scheme consists of a fault diagnosis module and a reconfigurable controller module comprised of a baseline controller and two adaptive fault-tolerant controllers activated after fault detection and after fault isolation, respectively. Under certain assumptions, the closed-loop system's stability and leader-follower consensus properties are rigorously established under different modes of the FTC system, including the time-period before possible fault detection, between fault detection and possible isolation, and after fault isolation.

Yongcan Cao

One typical application of unmanned aerial vehicles is the intelligence, surveillance, and reconnaissance mission, where the objective is to improve situation awareness through information acquisition. For examples, an efficient way to gather information regarding a target is to deploy UAV in such a way that it orbits around this target at a desired distance. Such a UAV motion is called circumnavigation. The objective of the paper is to design a UAV control algorithm such that this circumnavigation mission is achieved under a GPS-denied environment using range-only measurement. The control algorithm is constructed in two steps. The first step is to design a UAV control algorithm by assuming the availability of both range and range rate measurements, where the associated control input is always bounded. The second step is to further eliminate the use of range rate measurement by using an estimated range rate, obtained via a sliding-mode estimator using range measurement, to replace actual range rate measurement. Such a controller design technique is applicable in the control design of other UAV navigation and control missions under a GPS-denied environment.

Yongcan Cao, Jonathan Muse, David Casbeer et al.

This paper presents two control algorithms enabling a UAV to circumnavigate an unknown target using range and range rate (i.e., the derivative of range) measurements. Given a prescribed orbit radius, both control algorithms (i) tend to drive the UAV toward the tangent of prescribed orbit when the UAV is outside or on the orbit, and (ii) apply zero control input if the UAV is inside the desired orbit. The algorithms differ in that, the first algorithm is smooth and unsaturated while the second algorithm is non-smooth and saturated. By analyzing properties associated with the bearing angle of the UAV relative to the target and through proper design of Lyapunov functions, it is shown that both algorithms produce the desired orbit for an arbitrary initial state. Three examples are provided as a proof of concept.