Zexiang Li

David Winkel, Niklas Strauß, Maximilian Bernhard et al.

Allocation tasks represent a class of problems where a limited amount of resources must be allocated to a set of entities at each time step. Prominent examples of this task include portfolio optimization or distributing computational workloads across servers. Allocation tasks are typically bound by linear constraints describing practical requirements that have to be strictly fulfilled at all times. In portfolio optimization, for example, investors may be obligated to allocate less than 30\% of the funds into a certain industrial sector in any investment period. Such constraints restrict the action space of allowed allocations in intricate ways, which makes learning a policy that avoids constraint violations difficult. In this paper, we propose a new method for constrained allocation tasks based on an autoregressive process to sequentially sample allocations for each entity. In addition, we introduce a novel de-biasing mechanism to counter the initial bias caused by sequential sampling. We demonstrate the superior performance of our approach compared to a variety of Constrained Reinforcement Learning (CRL) methods on three distinct constrained allocation tasks: portfolio optimization, computational workload distribution, and a synthetic allocation benchmark. Our code is available at: https://github.com/niklasdbs/paspo

Yang You, Alex Schutz, Zhikun Li et al.

Many high-level multi-agent planning problems, including multi-robot navigation and path planning, can be effectively modeled using deterministic actions and observations. In this work, we focus on such domains and introduce the class of Deterministic Decentralized POMDPs (Det-Dec-POMDPs). This is a subclass of Dec-POMDPs characterized by deterministic transitions and observations conditioned on the state and joint actions. We then propose a practical solver called Iterative Deterministic POMDP Planning (IDPP). This method builds on the classic Joint Equilibrium Search for Policies framework and is specifically optimized to handle large-scale Det-Dec-POMDPs that current Dec-POMDP solvers are unable to address efficiently.

Jinni Zhou, Hao Xu, Zexiang Li et al.

Tail-sitter vertical takeoff and landing (VTOL) unmanned aerial vehicles (UAVs) have the capability of hovering and performing efficient level flight with compact mechanical structures. We present a unified controller design for such UAVs, based on recurrent neural networks. An advantage of this design method is that the various flight modes (i.e., hovering, transition and level flight) of a VTOL UAV are controlled in a unified manner, as opposed to treating them separately and in the runtime switching one from another. The proposed controller consists of an outer-loop position controller and an inner-loop attitude controller. The inner-loop controller is composed of a proportional attitude controller and a loop-shaping linear angular rate controller. For the outer-loop controller, we propose a nonlinear solver to compute the desired attitude and thrust, based on the UAV dynamics and an aerodynamic model, in addition to a cascaded PID controller for the position and velocity tracking. We employ a recurrent neural network (RNN) to approximate the behavior of the nonlinear solver, which suffers from high computational complexity. The proposed RNN has negligible approximation errors, and can be implemented in real-time (e.g., 50 Hz). Moreover, the RNN generates much smoother outputs than the nonlinear solver. We provide an analysis of the stability and robustness of the overall closed-loop system. Simulation and experiments are also presented to demonstrate the effectiveness of the proposed method.

Xiaorui Zhu, Wenwu Zeng, Zexiang Li et al.

This paper presents a novel method to attenuate large horizontal wind disturbance for a small-scale unmanned autonomous helicopter combining wind tunnel-based experimental data and a backstepping algorithm. Large horizontal wind disturbance is harmful to autonomous helicopters, especially to small ones because of their low inertia and the high cross-coupling effects among the multiple inputs. In order to achieve more accurate and faster attenuation of large wind disturbance, a new hybrid control architecture is proposed to take advantage of the direct force/moment compensation based on the wind tunnel experimental data. In this architecture, large horizontal wind disturbance is treated as an additional input to the control system instead of a small perturbation around the equilibrium state. A backstepping algorithm is then designed to guarantee the stable convergence of the hilicopter to the desired position. The proposed technique is finally evaluated in simulation on the platform, HIROBO Eagle, compared with a traditional wind velocity compensation method.