Konstantin Kondak

Jongseok Lee, Ribin Balachandran, Konstantin Kondak et al.

This article presents a novel telepresence system for advancing aerial manipulation in dynamic and unstructured environments. The proposed system not only features a haptic device, but also a virtual reality (VR) interface that provides real-time 3D displays of the robot's workspace as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors namely a LiDAR, cameras and IMUs are utilized. For processing of the acquired sensory data, pose estimation pipelines are devised for industrial objects of both known and unknown geometries. We further propose an active learning pipeline in order to increase the sample efficiency of a pipeline component that relies on Deep Neural Networks (DNNs) based object detection. All these algorithms jointly address various challenges encountered during the execution of perception tasks in industrial scenarios. In the experiments, exhaustive ablation studies are provided to validate the proposed pipelines. Methodologically, these results commonly suggest how an awareness of the algorithms' own failures and uncertainty (`introspection') can be used tackle the encountered problems. Moreover, outdoor experiments are conducted to evaluate the effectiveness of the overall system in enhancing aerial manipulation capabilities. In particular, with flight campaigns over days and nights, from spring to winter, and with different users and locations, we demonstrate over 70 robust executions of pick-and-place, force application and peg-in-hole tasks with the DLR cable-Suspended Aerial Manipulator (SAM). As a result, we show the viability of the proposed system in future industrial applications.

Jongseok Lee, Ribin Balachandran, Harsimran Singh et al.

Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.

Jongseok Lee, Ribin Balachandran, Yuri S. Sarkisov et al.

This paper presents a novel telepresence system for enhancing aerial manipulation capabilities. It involves not only a haptic device, but also a virtual reality that provides a 3D visual feedback to a remotely-located teleoperator in real-time. We achieve this by utilizing onboard visual and inertial sensors, an object tracking algorithm and a pre-generated object database. As the virtual reality has to closely match the real remote scene, we propose an extension of a marker tracking algorithm with visual-inertial odometry. Both indoor and outdoor experiments show benefits of our proposed system in achieving advanced aerial manipulation tasks, namely grasping, placing, force exertion and peg-in-hole insertion.

Yuri S. Sarkisov, Min Jun Kim, Andre Coelho et al.

This paper presents a design of oscillation damping control for the cable-Suspended Aerial Manipulator (SAM). The SAM is modeled as a double pendulum, and it can generate a body wrench as a control action. The main challenge is the fact that there is only one onboard IMU sensor which does not provide full information on the system state. To overcome this difficulty, we design a controller motivated by a simplified SAM model. The proposed controller is very simple yet robust to model uncertainties. Moreover, we propose a gain tuning rule by formulating the proposed controller in the form of output feedback linear quadratic regulation problem. Consequently, it is possible to quickly dampen oscillations with minimal energy consumption. The proposed approach is validated through simulations and experiments.

Andre Coelho, Harsimran Singh, Tin Muskardin et al.

Despite being one of the most robust methods in bilateral teleoperation, Time Domain Passivity Approach (TDPA) presents the drawback of accumulating position drift between master and slave devices. The lack of position synchronization poses an obstacle to the performance of teleoperation and may prevent the successful accomplishment of such tasks. Several techniques have been developed in order to solve the position-drift problem in TDPA-based teleoperation. However, they either present poor transparency by over-conservatively constraining force feedback or add high impulse-like force signals that can be harmful to the hardware and to the human operator. We propose a new approach to compensate position drift in TDPA-based teleoperation in a smoother way, which keeps the forces within the normal range of the teleoperation task while preserving the level of transparency and the robust stability of energy-based TDPA. We also add a way of tuning the compensator to behave in accordance with the task being performed, whether it requires faster or smoother compensation. The feasibility and performance of the method were experimentally validated. Good position tracking and regular-amplitude forces are demonstrated with up to 500 ms round-trip constant and variable delays for hard-wall contacts.

Andre Coelho, Harsimran Singh, Konstantin Kondak et al.

Attaching a robotic manipulator to a flying base allows for significant improvements in the reachability and versatility of manipulation tasks. In order to explore such systems while taking advantage of human capabilities in terms of perception and cognition, bilateral teleoperation arises as a reasonable solution. However, since most telemanipulation tasks require visual feedback in addition to the haptic one, real-time (task-dependent) positioning of a video camera, which is usually attached to the flying base, becomes an additional objective to be fulfilled. Since the flying base is part of the kinematic structure of the robot, if proper care is not taken, moving the video camera could undesirably disturb the end-effector motion. For that reason, the necessity of controlling the base position in the null space of the manipulation task arises. In order to provide the operator with meaningful information about the limits of the allowed motions in the null space, this paper presents a novel haptic concept called Null-Space Wall. In addition, a framework to allow stable bilateral teleoperation of both tasks is presented. Numerical simulation data confirm that the proposed framework is able to keep the system passive while allowing the operator to perform time-delayed telemanipulation and command the base to a task-dependent optimal pose.

Andre Coelho, Christian Ott, Harsimran Singh et al.

When, in addition to stability, position synchronization is also desired in bilateral teleoperation, Time Domain Passivity Approach (TDPA) alone might not be able to fulfill the desired objective. This is due to an undesired effect caused by admittance type passivity controllers, namely position drift. Previous works focused on developing TDPA-based drift compensation methods to solve this issue. It was shown that, in addition to reducing drift, one of the proposed methods was able to keep the force signals within their normal range, guaranteeing the safety of the task. However, no multi-DoF treatment of those approaches has been addressed. In that scope, this paper focuses on providing an extension of previous TDPA-based approaches to multi-DoF Cartesian-space teleoperation. An analysis of the convergence properties of the presented method is also provided. In addition, its applicability to multi-DoF devices is shown through hardware experiments and numerical simulation with round-trip time delays up to 700 ms.

Evgenii Safronov, Michael Vilzmann, Dzmitry Tsetserukou et al.

Complex aircraft systems are becoming a target for automation. For successful operation, they require both efficient and readable mission execution system. Flight control computer (FCC) units, as well as all important subsystems, are often duplicated. Discrete nature of mission execution systems does not allow small differences in data flow among redundant FCCs which are acceptable for continuous control algorithms. Therefore, mission state consistency has to be specifically maintained. We present a novel mission execution system which includes FCC state synchronization. To achieve this result we developed a new concept of Asynchronous Behavior Tree with Memory and proposed a state synchronization algorithm. The implemented system was tested and proven to work in a real-time simulation of High Altitude Pseudo Satellite (HAPS) mission.

Yuri S. Sarkisov, Min Jun Kim, Davide Bicego et al.

High risk of a collision between rotor blades and the obstacles in a complex environment imposes restrictions on the aerial manipulators. To solve this issue, a novel system cable-Suspended Aerial Manipulator (SAM) is presented in this paper. Instead of attaching a robotic manipulator directly to an aerial carrier, it is mounted on an active platform which is suspended on the carrier by means of a cable. As a result, higher safety can be achieved because the aerial carrier can keep a distance from the obstacles. For self-stabilization, the SAM is equipped with two actuation systems: winches and propulsion units. This paper presents an overview of the SAM including the concept behind, hardware realization, control strategy, and the first experimental results.

Min Jun Kim, Jianjie Lin, Konstantin Kondak et al.

This paper presents an approach to damp out the oscillatory motion of the pendulum-like hanging platform on which a robotic manipulator is mounted. To this end, moving masses were installed on top of the platform. In this paper, asymptotic stability of the platform (which implies oscillation damping) is achieved by designing reference acceleration of the moving masses properly. A main feature of this work is that we can achieve asymptotic stability of not only the platform, but also the moving masses, which may be challenging due to the under-actuation nature. The proposed scheme is validated by the simulation studies.

Min Jun Kim, Ribin Balachandran, Marco De Stefano et al.

This paper presents a passive compliance control for aerial manipulators to achieve stable environmental interactions. The main challenge is the absence of actuation along body-planar directions of the aerial vehicle which might be required during the interaction to preserve passivity. The controller proposed in this paper guarantees passivity of the manipulator through a proper choice of end-effector coordinates, and that of vehicle fuselage is guaranteed by exploiting time domain passivity technique. Simulation studies validate the proposed approach.