Markku Suomalainen

Nikunj Arora, Markku Suomalainen, Matti Pouke et al.

We propose augmenting immersive telepresence by adding a virtual body, representing the user's own arm motions, as realized through a head-mounted display and a 360-degree camera. Previous research has shown the effectiveness of having a virtual body in simulated environments; however, research on whether seeing one's own virtual arms increases presence or preference for the user in an immersive telepresence setup is limited. We conducted a study where a host introduced a research lab while participants wore a head-mounted display which allowed them to be telepresent at the host's physical location via a 360-degree camera, either with or without a virtual body. We first conducted a pilot study of 20 participants, followed by a pre-registered 62 participant confirmatory study. Whereas the pilot study showed greater presence and preference when the virtual body was present, the confirmatory study failed to replicate these results, with only behavioral measures suggesting an increase in presence. After analyzing the qualitative data and modeling interactions, we suspect that the quality and style of the virtual arms, and the contrast between animation and video, led to individual differences in reactions to the virtual body which subsequently moderated feelings of presence.

Markku Suomalainen, Basak Sakcak, Adhi Widagdo et al.

We propose unwinding the rotations experienced by the user of an immersive telepresence robot to improve comfort and reduce VR sickness of the user. By immersive telepresence we refer to a situation where a 360\textdegree~camera on top of a mobile robot is streaming video and audio into a head-mounted display worn by a remote user possibly far away. Thus, it enables the user to be present at the robot's location, look around by turning the head and communicate with people near the robot. By unwinding the rotations of the camera frame, the user's viewpoint is not changed when the robot rotates. The user can change her viewpoint only by physically rotating in her local setting; as visual rotation without the corresponding vestibular stimulation is a major source of VR sickness, physical rotation by the user is expected to reduce VR sickness. We implemented unwinding the rotations for a simulated robot traversing a virtual environment and ran a user study (N=34) comparing unwinding rotations to user's viewpoint turning when the robot turns. Our results show that the users found unwound rotations more preferable and comfortable and that it reduced their level of VR sickness. We also present further results about the users' path integration capabilities, viewing directions, and subjective observations of the robot's speed and distances to simulated people and objects.

Markku Suomalainen, Yiannis Karayiannidis, Ville Kyrki

In this survey, we present the current status on robots performing manipulation tasks that require varying contact with the environment, such that the robot must either implicitly or explicitly control the contact force with the environment to complete the task. Robots can perform more and more manipulation tasks that are still done by humans, and there is a growing number of publications on the topics of 1) performing tasks that always require contact and 2) mitigating uncertainty by leveraging the environment in tasks that, under perfect information, could be performed without contact. The recent trends have seen robots perform tasks earlier left for humans, such as massage, and in the classical tasks, such as peg-in-hole, there is a more efficient generalization to other similar tasks, better error tolerance, and faster planning or learning of the tasks. Thus, in this survey we cover the current stage of robots performing such tasks, starting from surveying all the different in-contact tasks robots can perform, observing how these tasks are controlled and represented, and finally presenting the learning and planning of the skills required to complete these tasks.

Markku Suomalainen, Katherine J. Mimnaugh, Israel Becerra et al.

In this paper, we analyze how different path aspects affect a user's experience, mainly VR sickness and overall comfort, while immersed in an autonomously moving telepresence robot through a virtual reality headset. In particular, we focus on how the robot turns and the distance it keeps from objects, with the goal of planning suitable trajectories for an autonomously moving immersive telepresence robot in mind; rotational acceleration is known for causing the majority of VR sickness, and distance to objects modulates the optical flow. We ran a within-subjects user study (n = 36, women = 18) in which the participants watched three panoramic videos recorded in a virtual museum while aboard an autonomously moving telepresence robot taking three different paths varying in aspects such as turns, speeds, or distances to walls and objects. We found a moderate correlation between the users' sickness as measured by the SSQ and comfort on a 6-point Likert scale across all paths. However, we detected no association between sickness and the choice of the most comfortable path, showing that sickness is not the only factor affecting the comfort of the user. The subjective experience of turn speed did not correlate with either the SSQ scores or comfort, even though people often mentioned turning speed as a source of discomfort in the open-ended questions. Through exploring the open-ended answers more carefully, a possible reason is that the length and lack of predictability also play a large role in making people observe turns as uncomfortable. A larger subjective distance from walls and objects increased comfort and decreased sickness both in quantitative and qualitative data. Finally, the SSQ subscales and total weighted scores showed differences by age group and by gender.

Katherine J. Mimnaugh, Markku Suomalainen, Israel Becerra et al.

This paper considers how the motions of a telepresence robot moving autonomously affect a person immersed in the robot through a head-mounted display. In particular, we explore the preference, comfort, and naturalness of elements of piecewise linear paths compared to the same elements on a smooth path. In a user study, thirty-six subjects watched panoramic videos of three different paths through a simulated museum in virtual reality and responded to questionnaires regarding each path. Preference for a particular path was influenced the most by comfort, forward speed, and characteristics of the turns. Preference was also strongly associated with the users' perceived naturalness, which was primarily determined by the ability to see salient objects, the distance to the walls and objects, as well as the turns. Participants favored the paths that had a one meter per second forward speed and rated the path with the least amount of turns as the most comfortable

Katherine J. Mimnaugh, Markku Suomalainen, Israel Becerra et al.

This paper presents some early work and future plans regarding how the autonomous motions of a telepresence robot affect a person embodied in the robot through a head-mounted display. We consider the preferences, comfort, and the perceived naturalness of aspects of piecewise linear paths compared to the same aspects on a smooth path. In a user study, thirty-six subjects (eighteen females) watched panoramic videos of three different paths through a simulated museum in virtual reality and responded to questionnaires regarding each path. We found that comfort had a strong effect on path preference, and that the subjective feeling of naturalness also had a strong effect on path preference, even though people consider different things as natural. We describe a categorization of the responses regarding the naturalness of the robot's motion and provide a recommendation on how this can be applied more broadly. Although immersive robotic telepresence is increasingly being used for remote education, clinical care, and to assist people with disabilities or mobility complications, the full potential of this technology is limited by issues related to user experience. Our work addresses these shortcomings and will enable the future personalization of telepresence experiences for the improvement of overall remote communication and the enhancement of the feeling of presence in a remote location.

Shikha Sharma, Markku Suomalainen, Ville Kyrki

Compliant motions allow alignment of workpieces using naturally occurring interaction forces. However, free-floating objects do not have a fixed base to absorb the reaction forces caused by the interactions. Consequently, if the interaction forces are too high, objects can gain momentum and move away after contact. This paper proposes an approach based on direct force control for compliant manipulation of free-floating objects. The objective of the controller is to minimize the interaction forces while maintaining the contact. The proposed approach achieves this by maintaining small constant force along the motion direction and an apparent reduction of manipulator inertia along remaining Degrees of Freedom (DOF). Simulation results emphasize the importance of relative inertia of the robotic manipulator with respect to the free-floating object. The experiments were performed with KUKA LWR4+ manipulator arm and a two-dimensional micro-gravity emulator (object floating on an air bed), which was developed in-house. It was verified that the proposed control law is capable of controlling the interaction forces and aligning the tools without pushing the object away. We conclude that direct force control works better with a free-floating object than implicit force control algorithms, such as impedance control.

Israel Becerra, Markku Suomalainen, Eliezer Lozano et al.

This paper introduces an emerging motion planning problem by considering a human that is immersed into the viewing perspective of a remote robot. The challenge is to make the experience both effective (such as delivering a sense of presence) and comfortable (such as avoiding adverse sickness symptoms, including nausea). We refer to this challenging new area as human perception-optimized planning and propose a general multiobjective optimization framework that can be instantiated in many envisioned scenarios. We then consider a specific VR telepresence task as a case of human perception-optimized planning, in which we simulate a robot that sends 360 video to a remote user to be viewed through a head-mounted display. In this particular task, we plan trajectories that minimize VR sickness (and thereby maximize comfort). An A* type method is used to create a Pareto-optimal collection of piecewise linear trajectories while taking into account criteria that improve comfort. We conducted a study with human subjects touring a virtual museum, in which paths computed by our algorithm are compared against a reference RRT-based trajectory. Generally, users suffered less from VR sickness and preferred the paths created by the presented algorithm.

Markku Suomalainen, Alexandra Q. Nilles, Steven M. LaValle

This paper applies the principles of Virtual Reality (VR) to robots, rather than living organisms. A simulator, of either physical states or information states, renders outputs to custom displays that fool the robot's sensors. This enables a robot to experience a combination of real and virtual sensor inputs, combining the efficiency of simulation and the benefits of real world sensor inputs. Thus, the robot can be taken through targeted experiences that are more realistic than pure simulation, yet more feasible and controllable than pure real-world experiences. We define two distinctive methods for applying VR to robots, namely black box and white box; based on these methods we identify potential applications, such as testing and verification procedures that are better than simulation, the study of spoofing attacks and anti-spoofing techniques, and sample generation for machine learning. A general mathematical framework is presented, along with a simple experiment, detailed examples, and discussion of the implications.

Markku Suomalainen, Sylvain Calinon, Emmanuel Pignat et al.

In this paper we show how different choices regarding compliance affect a dual-arm assembly task. In addition, we present how the compliance parameters can be learned from a human demonstration. Compliant motions can be used in assembly tasks to mitigate pose errors originating from, for example, inaccurate grasping. We present analytical background and accompanying experimental results on how to choose the center of compliance to enhance the convergence region of an alignment task. Then we present the possible ways of choosing the compliant axes for accomplishing alignment in a scenario where orientation error is present. We show that an earlier presented Learning from Demonstration method can be used to learn motion and compliance parameters of an impedance controller for both manipulators. The learning requires a human demonstration with a single teleoperated manipulator only, easing the execution of demonstration and enabling usage of manipulators at difficult locations as well. Finally, we experimentally verify our claim that having both manipulators compliant in both rotation and translation can accomplish the alignment task with less total joint motions and in shorter time than moving one manipulator only. In addition, we show that the learning method produces the parameters that achieve the best results in our experiments.

Dennis Ehlers, Markku Suomalainen, Jens Lundell et al.

We present a Learning from Demonstration method for teaching robots to perform search strategies imitated from humans in scenarios where alignment tasks fail due to position uncertainty. The method utilizes human demonstrations to learn both a state invariant dynamics model and an exploration distribution that captures the search area covered by the demonstrator. We present two alternative algorithms for computing a search trajectory from the exploration distribution, one based on sampling and another based on deterministic ergodic control. We augment the search trajectory with forces learnt through the dynamics model to enable searching both in force and position domains. An impedance controller with superposed forces is used for reproducing the learnt strategy. We experimentally evaluate the method on a KUKA LWR4+ performing a 2D peg-in-hole and a 3D electricity socket task. Results show that the proposed method can, with only few human demonstrations, learn to complete the search task.

Markku Suomalainen, Fares J. Abu-Dakka, Ville Kyrki

We present a novel method for learning from demonstration 6-D tasks that can be modeled as a sequence of linear motions and compliances. The focus of this paper is the learning of a single linear primitive, many of which can be sequenced to perform more complex tasks. The presented method learns from demonstrations only, without any prior information, how to take advantage of mechanical gradients in in-contact tasks, such as assembly, both for translations and rotations. The method assumes there exists a desired linear direction in 6-D which, if followed by the manipulator, leads the robot's end-effector to the goal area shown in the demonstration, either in free space or by leveraging contact through compliance. First, demonstrations are gathered where the teacher explicitly shows the robot how the mechanical gradients can be used as guidance towards the goal. From the demonstrations, a set of directions is computed which would result in the observed motion at each timestep during a demonstration of a single primitive. By observing which direction is included in all these sets, we find a single desired direction which can reproduce the demonstrated motion. Finding the number of compliant axes and their directions in both rotation and translation is based on the assumption that in the presence of a desired direction of motion, all other observed motion is caused by the contact force of the environment, signalling the need for compliance. We evaluate the method on a KUKA LWR4+ robot with test setups imitating typical tasks where a human would use compliance to cope with positional uncertainty. Results show that the method can successfully learn and reproduce compliant motions by taking advantage of the geometry of the task, therefore reducing the need for localization accuracy.

Tesfamichael Marikos Hagos, Markku Suomalainen, Ville Kyrki

This paper proposes an approach for segmenting a task consisting of compliant motions into phases, learning a primitive for each segmented phase of the task, and reproducing the task by sequencing primitives online based on the learned model. As compliant motions can "probe" the environment, using the interaction between the robot and the environment to detect phase transitions can make the transitions less prone to positional errors. This intuition leads us to model a task with a non-homogeneous Hidden Markov Model (HMM), wherein hidden phase transition probabilities depend on the interaction with the environment (wrench measured by an F/T sensor). Expectation-maximization algorithm is employed in estimating the parameters of the HMM model. During reproduction, the phase changes of a task are detected online using the forward algorithm, with the parameters learned from demonstrations. Cartesian impedance controller parameters are learned from the demonstrations to reproduce each phase of the task. The proposed approach is studied with a KUKA LWR4+ arm in two setups. Experiments show that the method can successfully segment and reproduce a task consisting of compliant motions with one or more demonstrations, even when demonstrations do not have the same starting position and external forces occur from different directions. Finally, we demonstrate that the method can also handle rotational motions.

Markku Suomalainen, Janne Koivumäki, Santeri Lampinen et al.

This paper presents, for the first time, a method for learning in-contact tasks from a teleoperated demonstration with a hydraulic manipulator. Due to the use of extremely powerful hydraulic manipulator, a force-reflected bilateral teleoperation is the most reasonable method of giving a human demonstration. An advanced subsystem-dynamic-based control design framework, virtual decomposition control (VDC), is used to design a stability-guaranteed controller for the teleoperation system, while taking into account the full nonlinear dynamics of the master and slave manipulators. The use of fragile force/ torque sensor at the tip of the hydraulic slave manipulator is avoided by estimating the contact forces from the manipulator actuators' chamber pressures. In the proposed learning method, it is observed that a surface-sliding tool has a friction-dependent range of directions (between the actual direction of motion and the contact force) from which the manipulator can apply force to produce the sliding motion. By this intuition, an intersection of these ranges can be taken over a motion to robustly find a desired direction for the motion from one or more demonstrations. The compliant axes required to reproduce the motion can be found by assuming that all motions outside the desired direction is caused by the environment, signalling the need for compliance. Finally, the learning method is incorporated to a novel VDC-based impedance control method to learn compliant behaviour from teleoperated human demonstrations. Experiments with 2-DOF hydraulic manipulator with a 475kg payload demonstrate the suitability and effectiveness of the proposed method to perform learning from demonstration (LfD) with heavy-duty hydraulic manipulators.

Markku Suomalainen, Ville Kyrki

This paper proposes a method to learn from human demonstration compliant contact motions, which take advantage of interaction forces between workpieces to align them, even when contact force may occur from different directions on different instances of reproduction. To manage the uncertainty in unstructured conditions, the motions learned with our method can be reproduced with an impedance controller. Learning from Demonstration is used because the planning of compliant motions in 3-D is computationally intractable. The proposed method will learn an individual compliant motion, many of which can be combined to solve more complex tasks. The method is based on measuring simultaneously the direction of motion and the forces acting on the end-effector. From these measurements we construct a set of constraints for motion directions which, with correct compliance, result in the observed motion. Constraints from multiple demonstrations are projected into a 2-D angular coordinate system where their intersection is determined to find a set of feasible desired directions, of which a single motion direction is chosen. The work is based on the assumption that movement in directions other than the desired direction is caused by interaction forces. Using this assumption, we infer the number of compliant axes and, if required, their directions. Experiments with a KUKA LWR4+ show that our method can successfully reproduce motions which require taking advantage of the environment.