Julie A. Adams

Sven Behnke, Julie A. Adams, David Locke

The $10M ANA Avatar XPRIZE aimed to create avatar systems that can transport human presence to remote locations in real time. The participants of this multi-year competition developed robotic systems that allow operators to see, hear, and interact with a remote environment in a way that feels as if they are truly there. On the other hand, people in the remote environment were given the impression that the operator was present inside the avatar robot. At the competition finals, held in November 2022 in Long Beach, CA, USA, the avatar systems were evaluated on their support for remotely interacting with humans, exploring new environments, and employing specialized skills. This article describes the competition stages with tasks and evaluation procedures, reports the results, presents the winning teams' approaches, and discusses lessons learned.

Julie A. Adams, Joshua Hamell, Phillip Walker

An open research question has been whether a single human can supervise a true heterogeneous swarm of robots completing tasks in real world environments. A general concern is whether or not the human's workload will be taxed to the breaking point. The Defense Advanced Research Projects Agency's OFFsensive Swarm-Enabled Tactics program's field exercises that occurred at U.S. Army urban training sites provided the opportunity to understand the impact of achieving such swarm deployments. The Command and Control of Aggregate Swarm Tactics integrator team's swarm commander users the heterogeneous robot swarm to conduct relevant missions. During the final OFFSET program field exercise, the team collected objective and subjective metrics related to teh swarm commander's human performance. A multi-dimensional workload algorithm that estimates overall workload based on five components of workload was used to analyze the results. While the swarm commander's workload estimate did cross the overload threshold frequently, the swarm commander was able to successfully complete the missions, often under challenging operational conditions. The presented results demonstrate that a single human can deploy a swarm of 100 heterogeneous robots to conduct real-world missions.

Grace Diehl, Julie A. Adams

Applications, such as military and disaster response, can benefit from robotic collectives' ability to perform multiple cooperative tasks (e.g., surveillance, damage assessments) efficiently across a large spatial area. Coalition formation algorithms can potentially facilitate collective robots' assignment to appropriate task teams; however, most coalition formation algorithms were designed for smaller multiple robot systems (i.e., 2-50 robots). Collectives' scale and domain-relevant constraints (i.e., distribution, near real-time, minimal communication) make coalition formation more challenging. This manuscript identifies the challenges inherent to designing coalition formation algorithms for very large collectives (e.g., 1000 robots). A survey of multiple robot coalition formation algorithms finds that most are unable to transfer directly to collectives, due to the identified system differences; however, auctions and hedonic games may be the most transferable. A simulation-based evaluation of three auction and hedonic game algorithms, applied to homogeneous and heterogeneous collectives, demonstrates that there are collective compositions for which no existing algorithm is viable; however, the experimental results and literature survey suggest paths forward.

Grace Diehl, Julie A. Adams

Robotic collectives for military and disaster response applications require coalition formation algorithms to partition robots into appropriate task teams. Collectives' missions will often incorporate tasks that require multiple high-level robot behaviors or services, which coalition formation must accommodate. The highly dynamic and unstructured application domains also necessitate that coalition formation algorithms produce near optimal solutions (i.e., >95% utility) in near real-time (i.e., <5 minutes) with very large collectives (i.e., hundreds of robots). No previous coalition formation algorithm satisfies these requirements. An initial evaluation found that traditional auction-based algorithms' runtimes are too long, even though the centralized simulator incorporated ideal conditions unlikely to occur in real-world deployments (i.e., synchronization across robots and perfect, instantaneous communication). The hedonic game-based GRAPE algorithm can produce solutions in near real-time, but cannot be applied to multiple service collectives. This manuscript integrates GRAPE and a services model, producing GRAPE-S and Pair-GRAPE-S. These algorithms and two auction baselines were evaluated using a centralized simulator with up to 1000 robots, and via the largest distributed coalition formation simulated evaluation to date, with up to 500 robots. The evaluations demonstrate that auctions transfer poorly to distributed collectives, resulting in excessive runtimes and low utility solutions. GRAPE-S satisfies the target domains' coalition formation requirements, producing near optimal solutions in near real-time, and Pair-GRAPE-S more than satisfies the domain requirements, producing optimal solutions in near real-time. GRAPE-S and Pair-GRAPE-S are the first algorithms demonstrated to support near real-time coalition formation for very large, distributed collectives with multiple services.

Julian Fortune, Julie A. Adams, Jamison Heard

Demanding task environments (e.g., supervising a remotely piloted aircraft) require performing tasks quickly and accurately; however, periods of low and high operator workload can decrease task performance. Intelligent modulation of the system's demands and interaction modality in response to changes in operator workload state may increase performance by avoiding undesirable workload states. This system requires real-time estimation of each workload component (i.e., cognitive, physical, visual, speech, and auditory) to adapt the correct modality. Existing workload systems estimate multiple workload components post-hoc, but few estimate speech workload, or function in real-time. An algorithm to estimate speech workload and mitigate undesirable workload states in real-time is presented. An analysis of the algorithm's accuracy is presented, along with the results demonstrating the algorithm's generalizability across individuals and human-machine teaming paradigms. Real-time speech workload estimation is a crucial element towards developing adaptive human-machine systems.

Karina A. Roundtree, Jason R. Cody, Jennifer Leaf et al.

Collective robotic systems are biologically inspired and advantageous due to their apparent global intelligence and emergent behaviors. Many applications can benefit from the incorporation of collectives, including environmental monitoring, disaster response missions, and infrastructure support. Transparency research has primarily focused on how the design of the models, visualizations, and control mechanisms influence human-collective interactions. Traditionally most evaluations have focused only on one particular system design element, evaluating its respective transparency. This manuscript analyzed two models and visualizations to understand how the system design elements impacted human-collective interactions, to quantify which model and visualization combination provided the best transparency, and provide design guidance, based on remote supervision of collectives. The consensus decision-making and baseline models, as well as an individual agent and abstract visualizations, were analyzed for sequential best-of-n decision-making tasks involving four collectives, composed of 200 entities each. Both models and visualizations provided transparency and influenced human-collective interactions differently. No single combination provided the best transparency.

Jason R. Cody, Karina A. Roundtree, Julie A. Adams

Robotic collectives are large groups (at least 50) of locally sensing and communicating robots that encompass characteristics of swarms and colonies, whose emergent behaviors accomplish complex tasks. Future human-collective teams will extend the ability of operators to monitor, respond, and make decisions in disaster response, search and rescue, and environmental monitoring problems. This manuscript evaluates two collective best-of-n decision models for enabling collectives to identify and choose the highest valued target from a finite set of n targets. Two challenges impede the future use of human-collective shared decisions: 1) environmental bias reduces collective decision accuracy when poorer targets are easier to evaluate than higher quality targets, and 2) little is understood about shared human-collective decision making interaction strategies. The two evaluated collective best-of-n models include an existing insect colony decision model and an extended bias-reducing model that attempts to reduce environmental bias in order to improve accuracy. Collectives using these two strategies are compared independently and as members of human-collective teams. Independently, the extended model is slower than the original model, but the extended algorithm is 57% more accurate in decisions where the optimal option is more difficult to evaluate. Human-collective teams using the bias-reducing model require less operator influence and achieve 25% higher accuracy with difficult decisions, than the human-collective teams using the original model. Further, a novel human-collective interaction strategy enables operators to adjust collective autonomy while making multiple simultaneous decisions.

Karina A. Roundtree, Jason R. Cody, Jennifer Leaf et al.

Interest in collective robotic systems has increased rapidly due to the potential benefits that can be offered to operators, such as increased safety and support, who perform challenging tasks in high-risk environments. Human-collective transparency research has focused on how the design of the algorithms, visualizations, and control mechanisms influence human-collective behavior. Traditional collective visualizations have shown all of the individual entities composing a collective, which may become problematic as collectives scale in size and heterogeneity, and tasks become more demanding. Human operators can become overloaded with information, which will negatively affect their understanding of the collective's current state and overall behaviors, which can cause poor teaming performance. An analysis of visualization transparency and the derived visualization design guidance, based on remote supervision of collectives, are the primary contributions of this manuscript. The individual agent and abstract visualizations were analyzed for sequential best-of-n decision-making tasks involving four collectives, composed of 200 entities each. The abstract visualization provided better transparency by enabling operators with different individual differences and capabilities to perform relatively the same and promoted higher human-collective performance.

Jamison Heard, Julian Fortune, Julie A. Adams

Supervisory-based human-robot teams are deployed in various dynamic and extreme environments (e.g., space exploration). Achieving high task performance in such environments is critical, as a mistake may lead to significant monetary loss or human injury. Task performance may be augmented by adapting the supervisory interface's interactions or autonomy levels based on the human supervisor's workload level, as workload is related to task performance. Typical adaptive systems rely solely on the human's overall or cognitive workload state to select what adaptation strategy to implement; however, overall workload encompasses many dimensions (i.e., cognitive, physical, visual, auditory, and speech) called workload components. Selecting an appropriate adaptation strategy based on a complete human workload state (rather than a single workload dimension) may allow for more impactful adaptations that ensure high task performance. A Supervisory-Based Adaptive Human-Robot Teaming Architecture (SAHRTA) that selects an appropriate level of autonomy or system interaction based on a complete real-time multi-dimensional workload estimate and predicted future task performance is introduced. SAHRTA was shown to improve overall task performance in a physically expanded version of the NASA Multi-Attribute Task Battery.

Musad Haque, Connor McGowan, Yifan Guo et al.

Unraveling the nature of the communication model that governs which two individuals in a swarm interact with each other is an important line of inquiry in the collective behavior sciences. A number of models have been proposed in the biological swarm literature, with the leading models being the metric, topological, and visual models. The hypothesis evaluated in this manuscript is whether the choice of a communication model impacts the performance of a tasked artificial swarm. The biological models are used to design coordination algorithms for a simulated swarm, which are evaluated over a range of six swarm robotics tasks. Each task has an associated set of performance metrics that are used to evaluate how the communication models fare against each other. The general findings demonstrate that the communication model significantly affects the swarm's performance for individual tasks, and this result implies that the communication model-task pairing is an important consideration when designing artificial swarms. Further analysis of each tasks' performance metrics reveal instances in which pairwise considerations of model and one of the various experimental factors becomes relevant. The reported research demonstrates that the artificial swarm's task performance can be increased through the careful selection of a communications model.