Michael Schreiber

Andre Rochow, Max Schwarz, Michael Schreiber et al.

VR Facial Animation is necessary in applications requiring clear view of the face, even though a VR headset is worn. In our case, we aim to animate the face of an operator who is controlling our robotic avatar system. We propose a real-time capable pipeline with very fast adaptation for specific operators. In a quick enrollment step, we capture a sequence of source images from the operator without the VR headset which contain all the important operator-specific appearance information. During inference, we then use the operator keypoint information extracted from a mouth camera and two eye cameras to estimate the target expression and head pose, to which we map the appearance of a source still image. In order to enhance the mouth expression accuracy, we dynamically select an auxiliary expression frame from the captured sequence. This selection is done by learning to transform the current mouth keypoints into the source camera space, where the alignment can be determined accurately. We, furthermore, demonstrate an eye tracking pipeline that can be trained in less than a minute, a time efficient way to train the whole pipeline given a dataset that includes only complete faces, show exemplary results generated by our method, and discuss performance at the ANA Avatar XPRIZE semifinals.

Marius Beul, Max Schwarz, Jan Quenzel et al.

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 posed diverse challenges for unmanned aerial vehicles (UAVs). We present our four tailored UAVs, specifically developed for individual aerial-robot tasks of MBZIRC, including custom hardware- and software components. In Challenge 1, a target UAV is pursued using a high-efficiency, onboard object detection pipeline to capture a ball from the target UAV. A second UAV uses a similar detection method to find and pop balloons scattered throughout the arena. For Challenge 2, we demonstrate a larger UAV capable of autonomous aerial manipulation: Bricks are found and tracked from camera images. Subsequently, they are approached, picked, transported, and placed on a wall. Finally, in Challenge 3, our UAV autonomously finds fires using LiDAR and thermal cameras. It extinguishes the fires with an onboard fire extinguisher. While every robot features task-specific subsystems, all UAVs rely on a standard software stack developed for this particular and future competitions. We present our mostly open-source software solutions, including tools for system configuration, monitoring, robust wireless communication, high-level control, and agile trajectory generation. For solving the MBZIRC 2020 tasks, we advanced the state of the art in multiple research areas like machine vision and trajectory generation. We present our scientific contributions that constitute the foundation for our algorithms and systems and analyze the results from the MBZIRC competition 2020 in Abu Dhabi, where our systems reached second place in the Grand Challenge. Furthermore, we discuss lessons learned from our participation in this complex robotic challenge.

Philipp Allgeuer, Grzegorz Ficht, Hafez Farazi et al.

The use of standard platforms in the field of humanoid robotics can lower the entry barrier for new research groups, and accelerate research by the facilitation of code sharing. Numerous humanoid standard platforms exist in the lower size ranges of up to 60cm, but beyond that humanoid robots scale up quickly in weight and price, becoming less affordable and more difficult to operate, maintain and modify. The igus Humanoid Open Platform is an affordable, fully open-source platform for humanoid research. At 92cm, the robot is capable of acting in an environment meant for humans, and is equipped with enough sensors, actuators and computing power to support researchers in many fields. The structure of the robot is entirely 3D printed, leading to a lightweight and visually appealing design. This paper covers the mechanical and electrical aspects of the robot, as well as the main features of the corresponding open-source ROS software. At RoboCup 2016, the platform was awarded the first International HARTING Open Source Prize.

Philipp Allgeuer, Hafez Farazi, Grzegorz Ficht et al.

The use of standard robotic platforms can accelerate research and lower the entry barrier for new research groups. There exist many affordable humanoid standard platforms in the lower size ranges of up to 60cm, but larger humanoid robots quickly become less affordable and more difficult to operate, maintain and modify. The igus Humanoid Open Platform is a new and affordable, fully open-source humanoid platform. At 92cm in height, the robot is capable of interacting in an environment meant for humans, and is equipped with enough sensors, actuators and computing power to support researchers in many fields. The structure of the robot is entirely 3D printed, leading to a lightweight and visually appealing design. The main features of the platform are described in this article.

Philipp Allgeuer, Hafez Farazi, Michael Schreiber et al.

The use of standard platforms in the field of humanoid robotics can accelerate research, and lower the entry barrier for new research groups. While many affordable humanoid standard platforms exist in the lower size ranges of up to 60cm, beyond this the few available standard platforms quickly become significantly more expensive, and difficult to operate and maintain. In this paper, the igus Humanoid Open Platform is presented---a new, affordable, versatile and easily customisable standard platform for humanoid robots in the child-sized range. At 90cm, the robot is large enough to interact with a human-scale environment in a meaningful way, and is equipped with enough torque and computing power to foster research in many possible directions. The structure of the robot is entirely 3D printed, allowing for a lightweight and appealing design. The electrical and mechanical designs of the robot are presented, and the main features of the corresponding open-source ROS software are discussed. The 3D CAD files for all of the robot parts have been released open-source in conjunction with this paper.

Max Schwarz, Christian Lenz, Andre Rochow et al.

Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typical indoor scenarios, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR HMD worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying a synthesized image of the operator with facial animation, which enables direct interaction with remote persons. We evaluate our system successfully both in a user study with untrained operators as well as a longer and more complex integrated mission. We discuss lessons learned from the trials and possible improvements.

Jan Quenzel, Malte Splietker, Dmytro Pavlichenko et al.

Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal. We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow, potentially GNSS-restricted environments while maneuvering close to obstacles. Measurements from LiDAR and thermal cameras are fused to track fires, while relative navigation ensures successful extinguishing. We analyze and discuss our successful participation during the MBZIRC 2020, present further experiments, and provide insights into our lessons learned from the competition.

Christian Lenz, Jan Quenzel, Arul Selvam Periyasamy et al.

Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hard- and software components to address the challenge tasks of wall building and fire fighting are integrated into a fully autonomous system. The robot consists of a wheeled omnidirectional base, a 6 DoF manipulator arm equipped with a magnetic gripper, a highly efficient storage system to transport box-shaped objects, and a water spraying system to fight fires. The robot perceives its environment using 3D LiDAR as well as RGB and thermal camera-based perception modules, is capable of picking box-shaped objects and constructing a pre-defined wall structure, as well as detecting and localizing heat sources in order to extinguish potential fires. A high-level planner solves the challenge tasks using the robot skills. We analyze and discuss our successful participation during the MBZIRC 2020 finals, present further experiments, and provide insights to our lessons learned.

Christian Lenz, Max Schwarz, Andre Rochow et al.

Constructing large structures with robots is a challenging task with many potential applications that requires mobile manipulation capabilities. We present two systems for autonomous wall building that we developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Both systems autonomously perceive their environment, find bricks, and build a predefined wall structure. While the UGV uses a 3D LiDAR-based perception system which measures brick poses with high precision, the UAV employs a real-time camera-based system for visual servoing. We report results and insights from our successful participation at the MBZIRC 2020 Finals, additional lab experiments, and discuss the lessons learned from the competition.

Diego Rodriguez, Hafez Farazi, Grzegorz Ficht et al.

Individual and team capabilities are challenged every year by rule changes and the increasing performance of the soccer teams at RoboCup Humanoid League. For RoboCup 2019 in the AdultSize class, the number of players (2 vs. 2 games) and the field dimensions were increased, which demanded for team coordination and robust visual perception and localization modules. In this paper, we present the latest developments that lead team NimbRo to win the soccer tournament, drop-in games, technical challenges and the Best Humanoid Award of the RoboCup Humanoid League 2019 in Sydney. These developments include a deep learning vision system, in-walk kicks, step-based push-recovery, and team play strategies.

Max Schwarz, David Droeschel, Christian Lenz et al.

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state-of-the-art in autonomous operation of ground-based and flying robots. In this article, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally employed to turn a valve stem. Mario's omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium-sized objects in 3D laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem in order to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.

Max Schwarz, Christian Lenz, Germán Martín García et al.

Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a custom turntable capture system and transfer learning. It produces high-quality item segments, on which grasp poses are found. A planning component coordinates manipulation actions between two robot arms, minimizing execution time. The system has been demonstrated successfully at ARC, where our team reached second places in both the picking task and the final stow-and-pick task. We also evaluate individual components.

Max Schwarz, Tobias Rodehutskors, David Droeschel et al.

Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally and to step over obstacles or climb. Furthermore, we present advanced communication and teleoperation approaches, which include immersive 3D visualization, and 6D tracking of operator head and arm motions. The proposed system is evaluated in the DARPA Robotics Challenge, the DLR SpaceBot Cup Qualification and lab experiments. We also discuss the lessons learned from the competitions.

Hafez Farazi, Philipp Allgeuer, Grzegorz Ficht et al.

The trend in the RoboCup Humanoid League rules over the past few years has been towards a more realistic and challenging game environment. Elementary skills such as visual perception and walking, which had become mature enough for exciting gameplay, are now once again core challenges. The field goals are both white, and the walking surface is artificial grass, which constitutes a much more irregular surface than the carpet used before. In this paper, team NimbRo TeenSize, the winner of the TeenSize class of the RoboCup 2016 Humanoid League, presents its robotic platforms, the adaptations that had to be made to them, and the newest developments in visual perception and soccer behaviour.

Marcell Missura, Cedrick Münstermann, Philipp Allgeuer et al.

Over the past few years, soccer-playing humanoid robots have advanced significantly. Elementary skills, such as bipedal walking, visual perception, and collision avoidance have matured enough to allow for dynamic and exciting games. When two robots are fighting for the ball, they frequently push each other and balance recovery becomes crucial. In this paper, we report on insights we gained from systematic push experiments performed on a bipedal model and outline an online learning method we used to improve its push-recovery capabilities. In addition, we describe how the localization ambiguity introduced by the uniform goal color was resolved and report on the results of the RoboCup 2013 competition.

Max Schwarz, Julio Pastrana, Philipp Allgeuer et al.

In recent years, the introduction of affordable platforms in the KidSize class of the Humanoid League has had a positive impact on the performance of soccer robots. The lack of readily available larger robots, however, severely affects the number of participants in Teen- and AdultSize and consequently the progress of research that focuses on the challenges arising with robots of larger weight and size. This paper presents the first hardware release of a low cost Humanoid TeenSize open platform for research, the first software release, and the current state of ROS-based software development. The NimbRo-OP robot was designed to be easily manufactured, assembled, repaired, and modified. It is equipped with a wide-angle camera, ample computing power, and enough torque to enable full-body motions, such as dynamic bipedal locomotion, kicking, and getting up.

Grzegorz Ficht, Dmytro Pavlichenko, Philipp Allgeuer et al.

The ongoing evolution of the RoboCup Humanoid League led in 2017 to the introduction of one vs. one soccer games for the AdultSize robots, which motived our team NimbRo to enter this category. In this paper, we present the mechatronic design of our upgraded robot Copedo and the newly developed NimbRo-OP2, which received the RoboCup Design Award. We also describe improved approaches to visual perception of the game situation, including compassless localization on a soccer field with symmetric appearance, and the generation of soccer behaviors. At RoboCup 2017 in Nagoya, our robots played very well, winning the AdultSize soccer tournament with high scores. Our robots also won the technical challenges and we present the developed solutions.