Carmel Majidi

Akhil Padmanabha, Sonal Choudhary, Carmel Majidi et al. · cmu

An objective measurement of chronic itch is necessary for improvements in patient care for numerous medical conditions. While wearables have shown promise for scratch detection, they are currently unable to estimate scratch intensity, preventing a comprehensive understanding of the effect of itch on an individual. In this work, we present a framework for the estimation of scratch intensity in addition to the detection of scratch. This is accomplished with a multimodal ring device, consisting of an accelerometer and a contact microphone, a pressure-sensitive tablet for capturing ground truth intensity values, and machine learning algorithms for regression of scratch intensity on a 0-600 milliwatts (mW) power scale that can be mapped to a 0-10 continuous scale. We evaluate the performance of our algorithms on 20 individuals using leave one subject out cross-validation and using data from 14 additional participants, we show that our algorithms achieve clinically-relevant discrimination of scratching intensity levels. By doing so, our device enables the quantification of the substantial variations in the interpretation of the 0-10 scale frequently utilized in patient self-reported clinical assessments. This work demonstrates that a finger-worn device can provide multidimensional, objective, real-time measures for the action of scratching.

Jeong Hun Lee, Junzhe Hu, Sofia Kwok et al.

Matching the swimming efficiency and agility of fish has remained an elusive goal in underwater robotics. Such locomotion capabilities rely on complex vortex interactions between the robot's body and the surrounding fluid. However, simulating these dynamics, which are governed by coupled ordinary and partial differential equations, is significantly more difficult than the multi-body dynamics of classical rigid robotic systems. We present a differentiable framework for simulating strongly coupled fluid-robot multiphysics as a unified optimization problem. The coupled manipulator and incompressible Navier-Stokes equations are derived together from a single Lagrangian using the principle of least action. We employ discrete variational mechanics to derive a stable, well-conditioned, and physically accurate scheme for jointly simulating articulated bodies and the surrounding fluid. We leverage the implicit function theorem to compute derivatives of the fully coupled dynamics. Using this simulator and its gradients, we realize undulating swimming gaits and optimize a highly dynamic C-start escape maneuver for a bioinspired eel robot. We validate both gaits on physical hardware, demonstrating successful sim-to-real transfer. Simulation code, hardware data, and schematics for the eel robot can be found here: https://unified-fluid-robot-multiphysics.github.io/

Raunaq Bhirangi, Abigail DeFranco, Jacob Adkins et al.

High cost and lack of reliability has precluded the widespread adoption of dexterous hands in robotics. Furthermore, the lack of a viable tactile sensor capable of sensing over the entire area of the hand impedes the rich, low-level feedback that would improve learning of dexterous manipulation skills. This paper introduces an inexpensive, modular, robust, and scalable platform -- the DManus -- aimed at resolving these challenges while satisfying the large-scale data collection capabilities demanded by deep robot learning paradigms. Studies on human manipulation point to the criticality of low-level tactile feedback in performing everyday dexterous tasks. The DManus comes with ReSkin sensing on the entire surface of the palm as well as the fingertips. We demonstrate effectiveness of the fully integrated system in a tactile aware task -- bin picking and sorting. Code, documentation, design files, detailed assembly instructions, trained models, task videos, and all supplementary materials required to recreate the setup can be found on https://sites.google.com/view/roboticsbenchmarks/platforms/dmanus.

Kiyn Chin, Carmel Majidi, Abhinav Gupta

Performing long-term experimentation or large-scale data collection for machine learning in the field of soft robotics is challenging, due to the hardware robustness and experimental flexibility required. In this work, we propose a modular parallel robotic manipulation platform suitable for such large-scale data collection and compatible with various soft-robotic fabrication methods. Considering the computational and theoretical difficulty of replicating the high-fidelity, faster-than-real-time simulations that enable large-scale data collection in rigid robotic systems, a robust soft-robotic hardware platform becomes a high priority development task for the field. The platform's modules consist of a pair of off-the-shelf electrical motors which actuate a customizable finger consisting of a compliant parallel structure. The parallel mechanism of the finger can be as simple as a single 3D-printed urethane or molded silicone bulk structure, due to the motors being able to fully actuate a passive structure. This design flexibility allows experimentation with soft mechanism varied geometries, bulk properties and surface properties. Additionally, while the parallel mechanism does not require separate electronics or additional parts, these can be included, and it can be constructed using multi-functional soft materials to study compatible soft sensors and actuators in the learning process. In this work, we validate the platform's ability to be used for policy gradient reinforcement learning directly on hardware in a benchmark 2D manipulation task. We additionally demonstrate compatibility with multiple fingers and characterize the design constraints for compatible extensions.

Akhil Padmanabha, Jessie Yuan, Janavi Gupta et al. · cmu

Physically assistive robots present an opportunity to significantly increase the well-being and independence of individuals with motor impairments or other forms of disability who are unable to complete activities of daily living. Speech interfaces, especially ones that utilize Large Language Models (LLMs), can enable individuals to effectively and naturally communicate high-level commands and nuanced preferences to robots. Frameworks for integrating LLMs as interfaces to robots for high level task planning and code generation have been proposed, but fail to incorporate human-centric considerations which are essential while developing assistive interfaces. In this work, we present a framework for incorporating LLMs as speech interfaces for physically assistive robots, constructed iteratively with 3 stages of testing involving a feeding robot, culminating in an evaluation with 11 older adults at an independent living facility. We use both quantitative and qualitative data from the final study to validate our framework and additionally provide design guidelines for using LLMs as speech interfaces for assistive robots. Videos and supporting files are located on our project website: https://sites.google.com/andrew.cmu.edu/voicepilot/

Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman et al.

Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics. These problems often involve using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to predict sequences of desirable physical quantities (e.g. force, inertial measurements). While classical approaches are powerful for locally-linear prediction problems, they often fall short when using real-world sensors. These sensors are typically non-linear, are affected by extraneous variables (e.g. vibration), and exhibit data-dependent drift. For many problems, the prediction task is exacerbated by small labeled datasets since obtaining ground-truth labels requires expensive equipment. In this work, we present Hierarchical State-Space Models (HiSS), a conceptually simple, new technique for continuous sequential prediction. HiSS stacks structured state-space models on top of each other to create a temporal hierarchy. Across six real-world sensor datasets, from tactile-based state prediction to accelerometer-based inertial measurement, HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments further indicate that HiSS demonstrates efficient scaling to smaller datasets and is compatible with existing data-filtering techniques. Code, datasets and videos can be found on https://hiss-csp.github.io.

Jessie Yuan, Janavi Gupta, Akhil Padmanabha et al. · cmu

Physically assistive robots present an opportunity to significantly increase the well-being and independence of individuals with motor impairments or other forms of disability who are unable to complete activities of daily living (ADLs). Speech interfaces, especially ones that utilize Large Language Models (LLMs), can enable individuals to effectively and naturally communicate high-level commands and nuanced preferences to robots. In this work, we demonstrate an LLM-based speech interface for a commercially available assistive feeding robot. Our system is based on an iteratively designed framework, from the paper "VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots," that incorporates human-centric elements for integrating LLMs as interfaces for robots. It has been evaluated through a user study with 11 older adults at an independent living facility. Videos are located on our project website: https://sites.google.com/andrew.cmu.edu/voicepilot/.

Guo Ning Sue, Yogita Choudhary, Richard Desatnik et al.

Ensuring safety via safety filters in real-world robotics presents significant challenges, particularly when the system dynamics is complex or unavailable. To handle this issue, learning-based safety filters recently gained popularity, which can be classified as model-based and model-free methods. Existing model-based approaches requires various assumptions on system model (e.g., control-affine), which limits their application in complex systems, and existing model-free approaches need substantial modifications to standard RL algorithms and lack versatility. This paper proposes a simple, plugin-and-play, and effective model-free safety filter learning framework. We introduce a novel reward formulation and use Q-learning to learn Q-value functions to safeguard arbitrary task specific nominal policies via filtering out their potentially unsafe actions. The threshold used in the filtering process is supported by our theoretical analysis. Due to its model-free nature and simplicity, our framework can be seamlessly integrated with various RL algorithms. We validate the proposed approach through simulations on double integrator and Dubin's car systems and demonstrate its effectiveness in real-world experiments with a soft robotic limb.

Akhil Padmanabha, Arpit Agarwal, Catherine Li et al.

Three-dimensional (3-D) skin surface reconstruction offers promise for objective and quantitative dermatological assessment, but no portable, high-resolution device exists that has been validated and used for depth reconstruction across various body locations. We present a compact 3-D skin reconstruction probe based on GelSight tactile imaging with a custom elastic gel and a learning-based reconstruction algorithm for micron-level wrinkle height estimation. Our probe, integrated into a handheld probe with force sensing for consistent contact, achieves a mean absolute error of 12.55 micron on wrinkle-like test objects. In a study with 15 participants without skin disorders, we provide the first validated wrinkle depth metrics across multiple body regions. We further demonstrate statistically significant reductions in wrinkle height at three locations following over-the-counter moisturizer application. Our work offers a validated tool for clinical and cosmetic skin analysis, with potential applications in diagnosis, treatment monitoring, and skincare efficacy evaluation.

Andrew P. Sabelhaus, Rohan K. Mehta, Anthony T. Wertz et al.

Untethered soft robots that locomote using electrothermally-responsive materials like shape memory alloy (SMA) face challenging design constraints for sensing actuator states. At the same time, modeling of actuator behaviors faces steep challenges, even with available sensor data, due to complex electrical-thermal-mechanical interactions and hysteresis. This article proposes a framework for in-situ sensing and dynamics modeling of actuator states, particularly temperature of SMA wires, which is used to predict robot motions. A planar soft limb is developed, actuated by a pair of SMA coils, that includes compact and robust sensors for temperature and angular deflection. Data from these sensors are used to train a neural network based on the long short-term memory (LSTM) architecture to model both unidirectional (single SMA) and bidirectional (both SMAs) motion. Predictions from the model demonstrate that data from the temperature sensor, combined with control inputs, allow for dynamics predictions over extraordinarily long open-loop timescales (10 minutes) with little drift. Prediction errors are on the order of the soft deflection sensor's accuracy. This architecture allows for compact designs of electrothermally-actuated soft robots that include sensing sufficient for motion predictions, helping to bring these robots into practical application.

Raunaq Bhirangi, Tess Hellebrekers, Carmel Majidi et al.

Soft sensors have continued growing interest in robotics, due to their ability to enable both passive conformal contact from the material properties and active contact data from the sensor properties. However, the same properties of conformal contact result in faster deterioration of soft sensors and larger variations in their response characteristics over time and across samples, inhibiting their ability to be long-lasting and replaceable. ReSkin is a tactile soft sensor that leverages machine learning and magnetic sensing to offer a low-cost, diverse and compact solution for long-term use. Magnetic sensing separates the electronic circuitry from the passive interface, making it easier to replace interfaces as they wear out while allowing for a wide variety of form factors. Machine learning allows us to learn sensor response models that are robust to variations across fabrication and time, and our self-supervised learning algorithm enables finer performance enhancement with small, inexpensive data collection procedures. We believe that ReSkin opens the door to more versatile, scalable and inexpensive tactile sensation modules than existing alternatives.

Zach J. Patterson, Andrew P. Sabelhaus, Carmel Majidi

Control of soft robotic manipulators remains a challenge for designs with advanced capabilities and novel actuation. Two significant limitations are multi-axis, three-dimensional motion of soft bodies alongside actuator dynamics and constraints, both of which are present in shape-memory-alloy (SMA)-powered soft robots. This article addresses both concerns with a robust feedback control scheme, demonstrating state tracking control for a soft robot manipulator of this type. Our controller uses a static beam bending model to approximate the soft limb as an LTI system, alongside a singular-value-decomposition compensator approach to decouple the multi-axial motion and an anti-windup element for the actuator saturation. We prove stability and verify robustness of our controller, with robustness intended to account for the unmodeled dynamics. Our implementation is verified in hardware tests of a soft SMA-powered limb, showing low tracking error, with promising results for future multi-limbed robots.

Anthony Wertz, Andrew P. Sabelhaus, Carmel Majidi

Practical use of robotic manipulators made from soft materials requires generating and executing complex motions. We present the first approach for generating trajectories of a thermally-actuated soft robotic manipulator. Based on simplified approximations of the soft arm and its antagonistic shape-memory alloy actuator coils, we justify a dynamics model of a discretized rigid manipulator with joint torques proportional to wire temperature. Then, we propose a method to calibrate this model from experimental data and demonstrate that the simulation aligns well with a hardware test. Finally, we use a direct collocation optimization with the robot's nonlinear dynamics to generate feasible state-input trajectories from a desired reference. Three experiments validate our approach for a single-segment robot in hardware: first using a hand-derived reference trajectory, then with two teach-and-repeat tests. The results show promise for both open-loop motion generation as well as for future applications with feedback.

Zach J. Patterson, Andrew P. Sabelhaus, Keene Chin et al.

Soft robots are capable of inherently safer interactions with their environment than rigid robots since they can mechanically deform in response to unanticipated stimuli. However, their complex mechanics can make planning and control difficult, particularly with tasks such as locomotion. In this work, we present a mobile and untethered underwater crawling soft robot, PATRICK, paired with a testbed that demonstrates closed-loop locomotion planning. PATRICK is inspired by the brittle star, with five flexible legs actuated by a total of 20 shape-memory alloy (SMA) wires, providing a rich variety of possible motions via its large input space. We propose a motion planning infrastructure based on a simple set of PATRICK's motion primitives, and provide experiments showing that the planner can command the robot to locomote to a goal state. These experiments contribute the first examples of closed-loop, state-space goal seeking of an underwater, untethered, soft crawling robot, and make progress towards full autonomy of soft mobile robotic systems.