Birgitta Dresp-Langley

Rongrong Liu, John M. Wandeto, Florent Nageotte et al.

This paper builds on our previous work by exploiting Artificial Intelligence to predict individual grip force variability in manual robot control. Grip forces were recorded from various loci in the dominant and non dominant hands of individuals by means of wearable wireless sensor technology. Statistical analyses bring to the fore skill specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain inspired neural network model that uses the output metric of a Self Organizing Map with unsupervised winner take all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time t and reliably captures the differences between novice and expert performance in terms of grip force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip force monitoring in real time to permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot interaction in environmental contexts of high sensory uncertainty. Parsimonious Artificial Intelligence (AI) assistance will contribute to the outcome of new types of surgery, in particular single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single Incision Laparoscopic Surgery).

Birgitta Dresp-Langley

This review explores biologically inspired learning as a model for intelligent robot control and sensing technology on the basis of specific examples. Hebbian synaptic learning is discussed as a functionally relevant model for machine learning and intelligence, as explained on the basis of examples from the highly plastic biological neural networks of invertebrates and vertebrates. Its potential for adaptive learning and control without supervision, the generation of functional complexity, and control architectures based on self organization is brought forward. Learning without prior knowledge based on excitatory and inhibitory neural mechanisms accounts for the process through which survival or task relevant representations are either reinforced or suppressed. The basic mechanisms of unsupervised biological learning drive synaptic plasticity and adaptation for behavioral success in living brains with different levels of complexity. The insights collected here point toward the Hebbian model as a choice solution for intelligent robotics and sensor systems. Keywords: Hebbian learning, synaptic plasticity, neural networks, self organization, brain, reinforcement, sensory processing, robot control

Rongrong Liu, Birgitta Dresp-Langley

This concept paper draws from our previous research on individual grip force data collected from biosensors placed on specific anatomical locations in the dominant and non dominant hands of operators performing a robot assisted precision grip task for minimally invasive endoscopic surgery. The specificity of the robotic system on the one hand, and that of the 2D image guided task performed in a real world 3D space on the other, constrain the individual hand and finger movements during task performance in a unique way. Our previous work showed task specific characteristics of operator expertise in terms of specific grip force profiles, which we were able to detect in thousands of highly variable individual data. This concept paper is focused on two complementary data analysis strategies that allow achieving such a goal. In contrast with other sensor data analysis strategies aimed at minimizing variance in the data, it is in this case here necessary to decipher the meaning of the full extent of intra and inter individual variance in the sensor data by using the appropriate statistical analyses, as shown in the first part of this paper. Then, it is explained how the computation of individual spatio temporal grip force profiles permits detecting expertise specific differences between individual users. It is concluded that these two analytic strategies are complementary. They enable drawing meaning from thousands of biosensor data reflecting human grip performance and its evolution with training, while fully taking into account their considerable inter and intra individual variability.

Birgitta Dresp-Langley, Rongrong Liu, John M. Wandeto

Individual grip force profiling of bimanual simulator task performance of experts and novices using a robotic control device designed for endoscopic surgery permits defining benchmark criteria that tell true expert task skills from the skills of novices or trainee surgeons. Grip force variability in a true expert and a complete novice executing a robot assisted surgical simulator task reveal statistically significant differences as a function of task expertise. Here we show that the skill specific differences in local grip forces are predicted by the output metric of a Self Organizing neural network Map (SOM) with a bio inspired functional architecture that maps the functional connectivity of somatosensory neural networks in the primate brain.

Rongrong Liu, Florent Nageotte, Philippe Zanne et al.

Deep learning has provided new ways of manipulating, processing and analyzing data. It sometimes may achieve results comparable to, or surpassing human expert performance, and has become a source of inspiration in the era of artificial intelligence. Another subfield of machine learning named reinforcement learning, tries to find an optimal behavior strategy through interactions with the environment. Combining deep learning and reinforcement learning permits resolving critical issues relative to the dimensionality and scalability of data in tasks with sparse reward signals, such as robotic manipulation and control tasks, that neither method permits resolving when applied on its own. In this paper, we present recent significant progress of deep reinforcement learning algorithms, which try to tackle the problems for the application in the domain of robotic manipulation control, such as sample efficiency and generalization. Despite these continuous improvements, currently, the challenges of learning robust and versatile manipulation skills for robots with deep reinforcement learning are still far from being resolved for real world applications.

Birgitta Dresp-Langley

The human somatosensory cortex is intimately linked to other central brain functions such as vision, audition, mechanoreception, and motor planning and control. These links are established through brain learning, and display a considerable functional plasticity. This latter fulfills an important adaptive role and ensures, for example, that humans are able to reliably manipulate and control objects in the physical world under constantly changing conditions in their immediate sensory environment. Variations in human grip force are a direct reflection of this specific kind of functional plasticity. Data from preliminary experiments where wearable wireless sensor technology (sensor gloves) was exploited to measure human grip force variations under varying sensory input conditions (eyes open or shut, soft music or hard music during gripping) are discussed here to show the extent to which grip force sensing permits quantifying somatosensory brain interactions and their functional plasticity. Experiments to take this preliminary work further are suggested. Implications for robotics, in particular the development of end-effector robots for upper limb movement planning and control, are brought forward.

Rongrong Liu, Florent Nageotte, Philippe Zanne et al.

Wearable biosensor technology enables real-time, convenient, and continuous monitoring of users behavioral signals. Such include signals relative to body motion, body temperature, biological or biochemical markers, and individual grip forces, which are studied in this paper. A four step pick and drop image guided and robot assisted precision task has been designed for exploiting a wearable wireless sensor glove system. Individual spatio temporal grip forces are analyzed on the basis of thousands of individual sensor data, collected from different locations on the dominant and non-dominant hands of each of three users in ten successive task sessions. Statistical comparisons reveal specific differences between grip force profiles of the individual users as a function of task skill level (expertise) and time.

Birgitta Dresp-Langley, Ole Kristian Ekseth, Jan Fesl et al.

Detecting quality in large unstructured datasets requires capacities far beyond the limits of human perception and communicability and, as a result, there is an emerging trend towards increasingly complex analytic solutions in data science to cope with this problem. This new trend towards analytic complexity represents a severe challenge for the principle of parsimony or Occams Razor in science. This review article combines insight from various domains such as physics, computational science, data engineering, and cognitive science to review the specific properties of big data. Problems for detecting data quality without losing the principle of parsimony are then highlighted on the basis of specific examples. Computational building block approaches for data clustering can help to deal with large unstructured datasets in minimized computation time, and meaning can be extracted rapidly from large sets of unstructured image or video data parsimoniously through relatively simple unsupervised machine learning algorithms. Why we still massively lack in expertise for exploiting big data wisely to extract relevant information for specific tasks, recognize patterns, generate new information, or store and further process large amounts of sensor data is then reviewed; examples illustrating why we need subjective views and pragmatic methods to analyze big data contents are brought forward. The review concludes on how cultural differences between East and West are likely to affect the course of big data analytics, and the development of increasingly autonomous artificial intelligence aimed at coping with the big data deluge in the near future.

Birgitta Dresp-Langley, Marie Monfouga

Pierons and Chocholles seminal psychophysical work predicts that human response time to information relative to visual contrast and sound frequency decreases when contrast intensity or sound frequency increases. The goal of this study is to bring to the fore the ability of individuals to use visual contrast intensity and sound frequency in combination for faster perceptual decisions of relative depth in planar object configurations on the basis of physical variations in luminance contrast. Computer controlled images with two abstract patterns of varying contrast intensity, one on the left and one on the right, preceded or not by a pure tone of varying frequency, were shown to healthy young humans in controlled experimental sequences. Their task was to decide as quickly as possible which of two patterns, the left or the right one, in a given image appeared to stand out as if it were nearer in terms of apparent or subjective visual depth. The results show that the combinations of varying relative visual contrast with sounds of varying frequency exploited here produced an additive effect on choice response times in terms of facilitation, where a stronger visual contrast combined with a higher sound frequency produced shorter forced choice response times. This new effect is predicted by crossmodal audiovisual probability summation.

Birgitta Dresp-Langley, Florent Nageotte, Philippe Zanne et al.

Wearable sensor systems with transmitting capabilities are currently employed for the biometric screening of exercise activities and other performance data. Such technology is generally wireless and enables the noninvasive monitoring of signals to track and trace user behaviors in real time. Examples include signals relative to hand and finger movements or force control reflected by individual grip force data. As will be shown here, these signals directly translate into task, skill, and hand specific, dominant versus non dominant hand, grip force profiles for different measurement loci in the fingers and palm of the hand. The present study draws from thousands of such sensor data recorded from multiple spatial locations. The individual grip force profiles of a highly proficient left handed exper, a right handed dominant hand trained user, and a right handed novice performing an image guided, robot assisted precision task with the dominant or the non dominant hand are analyzed. The step by step statistical approach follows Tukeys detective work principle, guided by explicit functional assumptions relating to somatosensory receptive field organization in the human brain. Correlation analyses in terms of Person Product Moments reveal skill specific differences in covariation patterns in the individual grip force profiles. These can be functionally mapped to from global to local coding principles in the brain networks that govern grip force control and its optimization with a specific task expertise. Implications for the real time monitoring of grip forces and performance training in complex task user systems are brought forward.

Michel de Mathelin, Florent Nageotte, Philippe Zanne et al.

STRAS (Single access Transluminal Robotic Assistant for Surgeons) is a new robotic system for application to intraluminal surgical procedures. Preclinical testing of STRAS has recently permitted to demonstrate major advantages of the system in comparison with classic procedures. Benchmark methods permitting to establish objective criteria for expertise need to be worked out now to effectively train surgeons on this new system in the near future. STRAS consists of three cable driven subsystems, one endoscope serving as guide, and two flexible instruments. The flexible instruments have three degrees of freedom and can be teleoperated by a single user via two specially designed master interfaces. In this study here, small force sensors sewn into a wearable glove to ergonomically fit the master handles of the robotic system were employed for monitoring the forces applied by an expert and a trainee who was a complete novice during all the steps of surgical task execution in a simulator task, a four step pick and drop. Analysis of gripforce profiles is performed sensor by sensor to bring to the fore specific differences in handgrip force profiles in specific sensor locations on anatomically relevant parts of the fingers and hand controlling the master slave system.

Birgitta Dresp-Langley

Biosensors and wearable sensor systems with transmitting capabilities are currently developed and used for the monitoring of health data, exercise activities, and other performance data. Unlike conventional approaches, these devices enable convenient, continuous, and/or unobtrusive monitoring of user behavioral signals in real time. Examples include signals relative to body motion, body temperature, blood flow parameters and a variety of biological or biochemical markers and, as will be shown in this chapter here, individual grip force data that directly translate into spatiotemporal grip force profiles for different locations on the fingers and palm of the hand. Wearable sensor systems combine innovation in sensor design, electronics, data transmission, power management, and signal processing for statistical analysis, as will be further shown herein. The first section of this chapter will provide an overview of the current state of the art in grip force profiling to highlight important functional aspects to be considered. In the next section, the contribution of wearable sensor technology in the form of sensor glove systems for the real-time monitoring of surgical task skill evolution in novices training in a simulator task will be described on the basis of recent examples. In the discussion, advantages and limitations will be weighed against each other.

Birgitta Dresp-Langley, John M. Wandeto

Cellular and molecular imaging techniques and models have been developed to characterize single stages of viral proliferation after focal infection of cells in vitro. The fast and automatic classification of cell imaging data may prove helpful prior to any further comparison of representative experimental data to mathematical models of viral propagation in host cells. Here, we use computer generated images drawn from a reproduction of an imaging model from a previously published study of experimentally obtained cell imaging data representing progressive viral particle proliferation in host cell monolayers. Inspired by experimental time-based imaging data, here in this study viral particle increase in time is simulated by a one-by-one increase, across images, in black or gray single pixels representing dead or partially infected cells, and hypothetical remission by a one-by-one increase in white pixels coding for living cells in the original image model. The image simulations are submitted to unsupervised learning by a Self-Organizing Map (SOM) and the Quantization Error in the SOM output (SOM-QE) is used for automatic classification of the image simulations as a function of the represented extent of viral particle proliferation or cell recovery. Unsupervised classification by SOM-QE of 160 model images, each with more than three million pixels, is shown to provide a statistically reliable, pixel precise, and fast classification model that outperforms human computer-assisted image classification by RGB image mean computation. The automatic classification procedure proposed here provides a powerful approach to understand finely tuned mechanisms in the infection and proliferation of virus in cell lines in vitro or other cells.

John M Wandeto, Birgitta Dresp-Langley

The quantization error in a fixed-size Self-Organizing Map (SOM) with unsupervised winner-take-all learning has previously been used successfully to detect, in minimal computation time, highly meaningful changes across images in medical time series and in time series of satellite images. Here, the functional properties of the quantization error in SOM are explored further to show that the metric is capable of reliably discriminating between the finest differences in local contrast intensities and contrast signs. While this capability of the QE is akin to functional characteristics of a specific class of retinal ganglion cells (the so-called Y-cells) in the visual systems of the primate and the cat, the sensitivity of the QE surpasses the capacity limits of human visual detection. Here, the quantization error in the SOM is found to reliably signal changes in contrast or colour when contrast information is removed from or added to the image, but not when the amount and relative weight of contrast information is constant and only the local spatial position of contrast elements in the pattern changes. While the RGB Mean reflects coarser changes in colour or contrast well enough, the SOM-QE is shown to outperform the RGB Mean in the detection of single-pixel changes in images with up to five million pixels. This could have important implications in the context of unsupervised image learning and computational building block approaches to large sets of image data (big data), including deep learning blocks, and automatic detection of contrast change at the nanoscale in Transmission or Scanning Electron Micrographs (TEM, SEM), or at the subpixel level in multispectral and hyper-spectral imaging data.

John Wandeto, Henry Nyongesa, Yves Remond et al.

Radiologists use time series of medical images to monitor the progression of a patient condition. They compare information gleaned from sequences of images to gain insight on progression or remission of the lesions, thus evaluating the progress of a patient condition or response to therapy. Visual methods of determining differences between one series of images to another can be subjective or fail to detect very small differences. We propose the use of quantization errors obtained from Self Organizing Maps for image content analysis. We tested this technique with MRI images to which we progressively added synthetic lesions. We have used a global approach that considers changes on the entire image as opposed to changes in segmented lesion regions only. We claim that this approach does not suffer from the limitations imposed by segmentation, which may compromise the results. Results show quantization errors increased with the increase in lesions on the images. The results are also consistent with previous studies using alternative approaches. We then compared the detectability ability of our method to that of human novice observers having to detect very small local differences in random-dot images. The quantization errors of the SOM outputs compared with correct positive rates, after subtraction of false positive rates (guess rates), increased noticeably and consistently with small increases in local dot size that were not detectable by humans. We conclude that our method detects very small changes in complex images and suggest that it could be implemented to assist human operators in image based decision making.

John M. Wandeto, Birgitta Dresp-Langley

Archiving large sets of medical or cell images in digital libraries may require ordering randomly scattered sets of image data according to specific criteria, such as the spatial extent of a specific local color or contrast content that reveals different meaningful states of a physiological structure, tissue, or cell in a certain order, indicating progression or recession of a pathology, or the progressive response of a cell structure to treatment. Here we used a Self Organized Map (SOM)-based, fully automatic and unsupervised, classification procedure described in our earlier work and applied it to sets of minimally processed grayscale and/or color processed Scanning Electron Microscopy (SEM) images of CD4+ T-lymphocytes (so-called helper cells) with varying extent of HIV virion infection. It is shown that the quantization error in the SOM output after training permits to scale the spatial magnitude and the direction of change (+ or -) in local pixel contrast or color across images of a series with a reliability that exceeds that of any human expert. The procedure is easily implemented and fast, and represents a promising step towards low-cost automatic digital image archiving with minimal intervention of a human operator.

Birgitta Dresp-Langley

Simulator training for image guided surgical interventions would benefit from intelligent systems that detect the evolution of task performance, and take control of individual speed precision strategies by providing effective automatic performance feedback. At the earliest training stages, novices frequently focus on getting faster at the task. This may, as shown here, compromise the evolution of their precision scores, sometimes irreparably, if it is not controlled for as early as possible. Artificial intelligence could help make sure that a trainee reaches optimal individual speed accuracy tradeoff by monitoring individual performance criteria, detecting critical trends at any given moment in time, and alerting the trainee as early as necessary when to slow down and focus on precision, or when to focus on getting faster. It is suggested that, for effective benchmarking, individual training statistics of novices are compared with the statistics of an expert surgeon. The speed accuracy functions of novices trained in a large number of experimental sessions reveal differences in individual speed versus precision strategies, and clarify why such strategies should be automatically detected and controlled for before further training on specific surgical task models, or clinical models, may be envisaged. How expert benchmark statistics may be exploited for automatic performance control is explained.

Birgitta Dresp-Langley

Gestalt theory has provided perceptual science with a conceptual framework which has inspired researchers ever since, taking the field of perceptual organization into the 21st century. This opinion article discusses the importance of rules of perceptual organization for the testing and design of visual interface technology. It is argued that major Gestalt principles, such as the law of good continuation or the principle of Praegnanz (suggested translation: salience), taken as examples here, are important to our understanding of visual image processing by a human observer. Perceptual integration of contrast information across collinear space, and the organization of objects in the 2D image plane into figure and ground are of a particular importance here. Visual interfaces for image-guided surgery illustrate the criticality of these two types of perceptual processes for reliable decision making and action. It is concluded that Gestalt theory continues to generate powerful concepts and insights for perceptual science placed within the context of major technological challenges of today.

Yasmine Boumenir, Fanny Georges, Jeremy Valentin et al.

Strategies for finding one's way through an unfamiliar environment may be helped by computer generated 2D maps, 3D virtual environments, or other navigation aids. The relative effectiveness of 2D and 3D virtual navigation aids was investigated. The wayfinding experiments (navigation tests) were conducted in a large, park-like environment. 24 participants (12 men, 12 women; age range = 22-50 years; M=32, SD = 7.4) were divided into three groups of four individuals per gender, who 1) explored a computer generated 2D map of the given route prior to navigation, 2) received a silent guided tour by means of an interactive 3D virtual representation, or 3) acquired direct experience of the real-world route through a silent guided tour where they were accompanied by a human individual who had expert knowledge of all the routes in the park. Participants from the different preparation groups then had to find the same route again on their own. 12 observers (six men and six women) were given a "simple" route with only one critical turn, and the other 12 a "complex" route with six critical turns. Navigation performances were compared with those of three experts who were highly familiar with all the routes of the park. Those among the naive participants who had benefitted from a direct experience (guided tour) prior to navigation, all found their way again on the simple and complex routes. Those who had explored the interactive 3D virtual environment were all unable to find their way on the complex route. The relative scale representation in the virtual 3D environment may have given incorrect impressions of relative distances between salient objects along the itinerary, rendering important landmark information useless.

Chiara Silvestri, Rene Motro, Bernard Maurin et al.

Conceptual design relies on extensive manipulation of morphological properties of real or virtual objects.This study investigates the nature of the perceptual information that could be retrieved from different representation modalities to reproduce structural properties of a complex object by drawing . The abstract and complex object (tensegrity simplex) was presented to two study populations (design experts/architects and non-experts) in three different representation modalities (2D image view explored visually only, digital 3D model explored visually using a computer mouse, the real object explored visually and manually. After viewing and exploring, observers had to draw the most critical parts of the structure by hand into a 2D reference frame. The results reveal a considerable performance advantage of digital 3D model exploration compared with real-world 3D object manipulation in the expert population.The results are discussed in terms of the specific nature of morphological cues made available through the different representation modalities.

A. U. Batmaz, M. de Mathelin, Birgitta Dresp-Langley

Effects of computer generated 2D and 3D views on the time and precision of bare-handed or tool-mediated eye-hand coordination were investigated in a pick-and-place-task with complete novices. All of them scored well above average in spatial perspective taking ability and performed the task with their dominant hand. Two groups of novices, four men and four women in each group, had to place a small object in a precise order on the centre of five targets on a Real-world Action Field (RAF), as swiftly as possible and as precisely as possible, using a tool or not (control). Each individual session consisted of four visual display conditions. The order of conditions was counterbalanced between individuals and sessions. Subjects looked at what their hands were doing 1) directly in front of them (natural top-down view) 2) in topdown 2D fisheye camera view 3) in top-down undistorted 2D view or 4) in 3D stereoscopic top-down view (head-mounted OCULUS DK 2). It was made sure that object movements in all image conditions matched the real-world movements in time and space. One group was looking at the 2D images with the monitor positioned sideways (sub-optimal); the other group was looking at the monitor placed straight ahead of them (near-optimal). All image viewing conditions had significantly detrimental effects on time (seconds) and precision (pixels) of task execution when compared with natural direct viewing.

AU Batmaz, M de Mathelin, Birgitta Dresp-Langley

In image-guided surgical tasks, the precision and timing of hand movements depend on the effectiveness of visual cues relative to specific target areas in the surgeons peri-personal space. Two-dimensional (2D) image views of real-world movements are known to negatively affect both constrained (with tool) and unconstrained(no tool) hand movements compared with direct action viewing. Task conditions where virtual 3D would generate and advantage for surgical eye-hand coordination are unclear. Here, we compared effects of 2D and 3D image views on the precision and timing of surgical hand movement trajectories in a simulator environment. Eight novices had to pick and place a small cube on target areas across different trajectory segments in the surgeons peri-personal space, with the dominant hand, with and without a tool, under conditions of: (1) direct (2) 2D fisheye camera and (3) virtual 3D viewing (headmounted). Significant effects of the location of trajectories in the surgeons peri-personal space on movement times and precision were found. Subjects were faster and more precise across specific target locations, depending on the viewing modality.

AU Batmaz, M de Mathelin, Birgitta Dresp-Langley

Background: The speed and precision with which objects are moved by hand or hand-tool interaction under image guidance depend on a specific type of visual and spatial sensorimotor learning. Novices have to learn to optimally control what their hands are doing in a real-world environment while looking at an image representation of the scene on a video monitor. Previous research has shown slower task execution times and lower performance scores under image-guidance compared with situations of direct action viewing. The cognitive processes for overcoming this drawback by training are not yet understood. Methods: We investigated the effects of training on the time and precision of direct view versus image guided object positioning on targets of a Real-world Action Field (RAF). Two men and two women had to learn to perform the task as swiftly and as precisely as possible with their dominant hand, using a tool or not and wearing a glove or not. Individuals were trained in sessions of mixed trial blocks with no feed-back. Results: As predicted, image-guidance produced significantly slower times and lesser precision in all trainees and sessionscompared with direct viewing. With training, all trainees get faster in all conditions, but only one of them gets reliably more precise in the image-guided conditions. Speed-accuracy trade-offs in the individual performance data show that the highest precision scores and steepest learning curve, for time and precision, were produced by the slowest starter.Conclusions: Performance evolution towards optimal precision is compromised when novices start by going as fast as they can. The findings have direct implications for individual skill monitoring in training programmes for image-guided technology applications with human operators.

John M. Wandeto, Henry O. Nyongesa, Birgitta Dresp-Langley

Time-series of satellite images may reveal important data about changes in environmental conditions and natural or urban landscape structures that are of potential interest to citizens, historians, or policymakers. We applied a fast method of image analysis using Self Organized Maps (SOM) and, more specifically, the quantization error (QE), for the visualization of critical changes in satellite images of Las Vegas, generated across the years 1984-2008, a period of major restructuration of the urban landscape. As shown in our previous work, the QE from the SOM output is a reliable measure of variability in local image contents. In the present work, we use statistical trend analysis to show how the QE from SOM run on specific geographic regions of interest extracted from satellite images can be exploited to detect both the magnitude and the direction of structural change across time at a glance. Significantly correlated demographic data for the same reference time period are highlighted. The approach is fast and reliable, and can be implemented for the rapid detection of potentially critical changes in time series of large bodies of image data.

Birgitta Dresp-Langley, John Mwangi Wandeto

The quantization error (QE) from SOM applied on time series of spatial contrast images with variable relative amount of white and dark pixel contents, as in monochromatic medical images or satellite images, is proven a reliable indicator of potentially critical changes in image homogeneity. The QE is shown to increase linearly with the variability in spatial contrast contents across time when contrast intensity is kept constant.