Damian M. Lyons

Mohamed Rahouti, Damian Lyons, Senthil Kumar Jagatheesaperumal et al.

Visual homing is a lightweight approach to visual navigation. Given the stored information of an initial 'home' location, the navigation task back to this location is achieved from any other location by comparing the stored home information to the current image and extracting a motion vector. A challenge that constrains the applicability of visual homing is that the home location must be within the robot's field of view to initiate the homing process. Thus, we propose a blockchain approach to visual navigation for a heterogeneous robot team over a wide area of visual navigation. Because it does not require map data structures, the approach is useful for robot platforms with a small computational footprint, and because it leverages current visual information, it supports a resilient and adaptive path selection. Further, we present a lightweight Proof-of-Work (PoW) mechanism for reaching consensus in the untrustworthy visual homing network.

Nick Trinh, Damian Lyons

Visual place recognition (VPR) is an important component technology for camera-based mapping and navigation applications. This is a challenging problem because images of the same place may appear quite different for reasons including seasonal changes, weather illumination, structural changes to the environment, as well as transient pedestrian or vehicle traffic. Papers focusing on generating image descriptors for VPR report their results using metrics such as recall@K and ROC curves. However, for a robot implementation, determining which matches are sufficiently good is often reduced to a manually set threshold. And it is difficult to manually select a threshold that will work for a variety of visual scenarios. This paper addresses the problem of automatically selecting a threshold for VPR by looking at the 'negative' Gaussian mixture statistics for a place - image statistics indicating not this place. We show that this approach can be used to select thresholds that work well for a variety of image databases and image descriptors.

Jason Hughes, Damian Lyons

An autonomous drone flying near obstacles needs to be able to detect and avoid the obstacles or it will collide with them. In prior work, drones can detect and avoid walls using data from camera, ultrasonic or laser sensors mounted either on the drone or in the environment. It is not always possible to instrument the environment, and sensors added to the drone consume payload and power - both of which are constrained for drones. This paper studies how data mining classification techniques can be used to predict where an obstacle is in relation to the drone based only on monitoring air-disturbance. We modeled the airflow of the rotors physically to deduce higher level features for classification. Data was collected from the drone's IMU while it was flying with a wall to its direct left, front and right, as well as with no walls present. In total 18 higher level features were produced from the raw data. We used an 80%, 20% train-test scheme with the RandomForest (RF), K-Nearest Neighbor (KNN) and GradientBoosting (GB) classifiers. Our results show that with the RF classifier and with 90% accuracy it can predict which direction a wall is in relation to the drone.

Kasper Grispino, Damian Lyons, Truong-Huy Nguyen

Human-to-human communications are enriched with affects and emotions, conveyed, and perceived through both verbal and nonverbal communication. It is our thesis that drone swarms can be used to communicate information enriched with effects via nonverbal channels: guiding, generally interacting with, or warning a human audience via their pattern of motions or behavior. And furthermore that this approach has unique advantages such as flexibility and mobility over other forms of user interface. In this paper, we present a user study to understand how human participants perceived and interpreted swarm behaviors of micro-drone Crazyflie quadcopters flying three different flight formations to bridge the psychological gap between front-end technologies (drones) and the human observers' emotional perceptions. We ask the question whether a human observer would in fact consider a swarm of drones in their immediate vicinity to be nonthreatening enough to be a vehicle for communication, and whether a human would intuit some communication from the swarm behavior, despite the lack of verbal or written language. Our results show that there is statistically significant support for the thesis that a human participant is open to interpreting the motion of drones as having intent and to potentially interpret their motion as communication. This supports the potential use of drone swarms as a communication resource, emergency guidance situations, policing of public events, tour guidance, etc.

D. M. Lyons, S. Zahra

It is important to be able to establish formal performance bounds for autonomous systems. However, formal verification techniques require a model of the environment in which the system operates; a challenge for autonomous systems, especially those expected to operate over longer timescales. This paper describes work in progress to automate the monitor and repair of ROS-based autonomous robot software written for an a-priori partially known and possibly incorrect environment model. A taint analysis method is used to automatically extract the data-flow sequence from input topic to publish topic, and instrument that code. A unique reinforcement learning approximation of MDP utility is calculated, an empirical and non-invasive characterization of the inherent objectives of the software designers. By comparing off-line (a-priori) utility with on-line (deployed system) utility, we show, using a small but real ROS example, that it's possible to monitor a performance criterion and relate violations of the criterion to parts of the software. The software is then patched using automated software repair techniques and evaluated against the original off-line utility.

Damian Lyons, James Finocchiaro, Michael Novitzky et al.

We propose a novel approach to the 'reality gap' problem, i.e., modifying a robot simulation so that its performance becomes more similar to observed real world phenomena. This problem arises whether the simulation is being used by human designers or in an automated policy development mechanism. We expect that the program/policy is developed using simulation, and subsequently deployed on a real system. We further assume that the program includes a monitor procedure with scalar output to determine when it is achieving its performance objectives. The proposed approach collects simulation and real world observations and builds conditional probability functions. These are used to generate paired roll-outs to identify points of divergence in behavior. These are used to generate {\it state-space kernels} that coerce the simulation into behaving more like observed reality. The method was evaluated using ROS/Gazebo for simulation and a heavily modified Traaxas platform in outdoor deployment. The results support not just that the kernel approach can force the simulation to behave more like reality, but that the modification is such that an improved control policy tested in the modified simulation also performs better in the real world.

Damian M. Lyons, Saba B. Zahra, Thomas M. Marshall

Large software systems often comprise programs written in different programming languages. In the case when cross-language interoperability is accomplished with a Foreign Function Interface (FFI), for example pybind11, Boost.Python, Emscripten, PyV8, or JNI, among many others, common software engineering tools, such as call-graph analysis, are obstructed by the opacity of the FFI. This complicates debugging and fosters potential inefficiency and security problems. One contributing issue is that there is little rigorous software design advice for multilingual software. In this paper, we present our progress towards a more rigorous design approach to multilingual software. The approach is based on the existing approach to the design of large-scale C++ systems developed by Lakos. The Lakosian approach is one of the few design methodologies to address physical design rather than just logical design. Using the MLSA toolkit developed in prior work for analysis of multilingual software, we focus in on one FFI -- the pybind11 FFI. An extension to the Lakosian C++ design rules is proposed to address multilingual software that uses pybind11. Using a sample of 50 public GitHub repositories that use pybind11, we measure how many repositories would currently satisfy these rules. We conclude with a proposed generalization of the pybind11-based rules for any multilingual software using an FFI interface.

Anne Marie Bogar, Damian M. Lyons, David Baird

Analysis of multilingual codebases is a topic of increasing importance. In prior work, we have proposed the MLSA (MultiLingual Software Analysis) architecture, an approach to the lightweight analysis of multilingual codebases, and have shown how it can be used to address the challenge of constructing a single call graph from multilingual software with mutual calls. This paper addresses the challenge of constructing monolingual call graphs in a lightweight manner (consistent with the objective of MLSA) which nonetheless yields sufficient information for resolving language interoperability calls. A novel approach is proposed which leverages information from a compiler-generated AST to provide the quality of call graph necessary, while the program itself is written using an Island Grammar that parses the AST providing the lightweight aspect necessary. Performance results are presented for a C/C++ implementation of the approach, PAIGE (Parsing AST using Island Grammar Call Graph Emitter) showing that despite its lightweight nature, it outperforms Doxgen, is robust to changes in the (Clang) AST, and is not restricted to C/C++.

Damian M. Lyons, Anne Marie Bogar, David Baird

Developer preferences, language capabilities and the persistence of older languages contribute to the trend that large software codebases are often multilingual, that is, written in more than one computer language. While developers can leverage monolingual software development tools to build software components, companies are faced with the problem of managing the resultant large, multilingual codebases to address issues with security, efficiency, and quality metrics. The key challenge is to address the opaque nature of the language interoperability interface: one language calling procedures in a second (which may call a third, or even back to the first), resulting in a potentially tangled, inefficient and insecure codebase. An architecture is proposed for lightweight static analysis of large multilingual codebases: the MLSA architecture. Its modular and table-oriented structure addresses the open-ended nature of multiple languages and language interoperability APIs. We focus here as an application on the construction of call-graphs that capture both inter-language and intra-language calls. The algorithms for extracting multilingual call-graphs from codebases are presented, and several examples of multilingual software engineering analysis are discussed. The state of the implementation and testing of MLSA is presented, and the implications for future work are discussed.

Trevor Buteau, Damian Lyons

Prior efforts to create an autonomous computer system capable of predicting what a human being is thinking or feeling from facial expression data have been largely based on outdated, inaccurate models of how emotions work that rely on many scientifically questionable assumptions. In our research, we are creating an empathetic system that incorporates the latest provable scientific understanding of emotions: that they are constructs of the human mind, rather than universal expressions of distinct internal states. Thus, our system uses a user-dependent method of analysis and relies heavily on contextual information to make predictions about what subjects are experiencing. Our system's accuracy and therefore usefulness are built on provable ground truths that prohibit the drawing of inaccurate conclusions that other systems could too easily make.

Truong-Huy D. Nguyen, Kasper Grispino, Damian Lyons

Drone swarms are teams of autonomous un-manned aerial vehicles that act as a collective entity. We are interested in humanizing drone swarms, equipping them with the ability to emotionally affect human users through their non-verbal motions. Inspired by recent findings in how observers are emotionally touched by watching dance moves, we investigate the questions of whether and how coordinated drone swarms' motions can achieve emotive impacts on general audience. Our preliminary study on Amazon Mechanical Turk led to a number of interesting findings, including both promising results and challenges.