Mohammad Khalid Jawed

Shivam K Panda, Yongkyu Lee, M. Khalid Jawed

The successful implementation of vision-based navigation in agricultural fields hinges upon two critical components: 1) the accurate identification of key components within the scene, and 2) the identification of lanes through the detection of boundary lines that separate the crops from the traversable ground. We propose Agronav, an end-to-end vision-based autonomous navigation framework, which outputs the centerline from the input image by sequentially processing it through semantic segmentation and semantic line detection models. We also present Agroscapes, a pixel-level annotated dataset collected across six different crops, captured from varying heights and angles. This ensures that the framework trained on Agroscapes is generalizable across both ground and aerial robotic platforms. Codes, models and dataset will be released at \href{https://github.com/shivamkumarpanda/agronav}{github.com/shivamkumarpanda/agronav}.

Qiaofeng Li, Tianyi Wang, Vwani Roychowdhury et al.

We present the interpretable meta neural ordinary differential equation (iMODE) method to rapidly learn generalizable (i.e., not parameter-specific) dynamics from trajectories of multiple dynamical systems that vary in their physical parameters. The iMODE method learns meta-knowledge, the functional variations of the force field of dynamical system instances without knowing the physical parameters, by adopting a bi-level optimization framework: an outer level capturing the common force field form among studied dynamical system instances and an inner level adapting to individual system instances. A priori physical knowledge can be conveniently embedded in the neural network architecture as inductive bias, such as conservative force field and Euclidean symmetry. With the learned meta-knowledge, iMODE can model an unseen system within seconds, and inversely reveal knowledge on the physical parameters of a system, or as a Neural Gauge to "measure" the physical parameters of an unseen system with observed trajectories. We test the validity of the iMODE method on bistable, double pendulum, Van der Pol, Slinky, and reaction-diffusion systems.

Andrew Choi, Mohammad Khalid Jawed, Jungseock Joo

As technology advances, the need for safe, efficient, and collaborative human-robot-teams has become increasingly important. One of the most fundamental collaborative tasks in any setting is the object handover. Human-to-robot handovers can take either of two approaches: (1) direct hand-to-hand or (2) indirect hand-to-placement-to-pick-up. The latter approach ensures minimal contact between the human and robot but can also result in increased idle time due to having to wait for the object to first be placed down on a surface. To minimize such idle time, the robot must preemptively predict the human intent of where the object will be placed. Furthermore, for the robot to preemptively act in any sort of productive manner, predictions and motion planning must occur in real-time. We introduce a novel prediction-planning pipeline that allows the robot to preemptively move towards the human agent's intended placement location using gaze and gestures as model inputs. In this paper, we investigate the performance and drawbacks of our early intent predictor-planner as well as the practical benefits of using such a pipeline through a human-robot case study.

Gyusam Chang, Tuan-Anh Vu, Vivek Alumootil et al.

While 3D Gaussian Splatting (3DGS) has rapidly advanced, its application in agriculture remains underexplored. Agricultural scenes present unique challenges for 3D reconstruction methods, particularly due to uneven illumination, occlusions, and a limited field of view. To address these limitations, we introduce \textbf{NIRPlant}, a novel multimodal dataset encompassing Near-Infrared (NIR) imagery, RGB imagery, textual metadata, Depth, and LiDAR data collected under varied indoor and outdoor lighting conditions. By integrating NIR data, our approach enhances robustness and provides crucial botanical insights that extend beyond the visible spectrum. Additionally, we leverage text-based metadata derived from vegetation indices, such as NDVI, NDWI, and the chlorophyll index, which significantly enriches the contextual understanding of complex agricultural environments. To fully exploit these modalities, we propose \textbf{NIRSplat}, an effective multimodal Gaussian splatting architecture employing a cross-attention mechanism combined with 3D point-based positional encoding, providing robust geometric priors. Comprehensive experiments demonstrate that \textbf{NIRSplat} outperforms existing landmark methods, including 3DGS, CoR-GS, and InstantSplat, highlighting its effectiveness in challenging agricultural scenarios. The code and dataset are publicly available at: https://github.com/StructuresComp/3D-Reconstruction-NIR

Yayun Du, Guofeng Zhang, Darren Tsang et al.

Smart weeding systems to perform plant-specific operations can contribute to the sustainability of agriculture and the environment. Despite monumental advances in autonomous robotic technologies for precision weed management in recent years, work on under-canopy weeding in fields is yet to be realized. A prerequisite of such systems is reliable detection and classification of weeds to avoid mistakenly spraying and, thus, damaging the surrounding plants. Real-time multi-class weed identification enables species-specific treatment of weeds and significantly reduces the amount of herbicide use. Here, our first contribution is the first adequately large realistic image dataset \textit{AIWeeds} (one/multiple kinds of weeds in one image), a library of about 10,000 annotated images of flax, and the 14 most common weeds in fields and gardens taken from 20 different locations in North Dakota, California, and Central China. Second, we provide a full pipeline from model training with maximum efficiency to deploying the TensorRT-optimized model onto a single board computer. Based on \textit{AIWeeds} and the pipeline, we present a baseline for classification performance using five benchmark CNN models. Among them, MobileNetV2, with both the shortest inference time and lowest memory consumption, is the qualified candidate for real-time applications. Finally, we deploy MobileNetV2 onto our own compact autonomous robot \textit{SAMBot} for real-time weed detection. The 90\% test accuracy realized in previously unseen scenes in flax fields (with a row spacing of 0.2-0.3 m), with crops and weeds, distortion, blur, and shadows, is a milestone towards precision weed control in the real world. We have publicly released the dataset and code to generate the results at \url{https://github.com/StructuresComp/Multi-class-Weed-Classification}.

Yongkyu Lee, Shivam Kumar Panda, Wei Wang et al.

We present Measure Anything, a comprehensive vision-based framework for dimensional measurement of objects with circular cross-sections, leveraging the Segment Anything Model (SAM). Our approach estimates key geometric features -- including diameter, length, and volume -- for rod-like geometries with varying curvature and general objects with constant skeleton slope. The framework integrates segmentation, mask processing, skeleton construction, and 2D-3D transformation, packaged in a user-friendly interface. We validate our framework by estimating the diameters of Canola stems -- collected from agricultural fields in North Dakota -- which are thin and non-uniform, posing challenges for existing methods. Measuring its diameters is critical, as it is a phenotypic traits that correlates with the health and yield of Canola crops. This application also exemplifies the potential of Measure Anything, where integrating intelligent models -- such as keypoint detection -- extends its scalability to fully automate the measurement process for high-throughput applications. Furthermore, we showcase its versatility in robotic grasping, leveraging extracted geometric features to identify optimal grasp points.

Harris Song, Tuan-Anh Vu, Sanjith Menon et al.

Detecting hidden or partially concealed objects remains a fundamental challenge in multimodal environments, where factors like occlusion, camouflage, and lighting variations significantly hinder performance. Traditional RGB-based detection methods often fail under such adverse conditions, motivating the need for more robust, modality-agnostic approaches. In this work, we present HiddenObject, a fusion framework that integrates RGB, thermal, and depth data using a Mamba-based fusion mechanism. Our method captures complementary signals across modalities, enabling enhanced detection of obscured or camouflaged targets. Specifically, the proposed approach identifies modality-specific features and fuses them in a unified representation that generalizes well across challenging scenarios. We validate HiddenObject across multiple benchmark datasets, demonstrating state-of-the-art or competitive performance compared to existing methods. These results highlight the efficacy of our fusion design and expose key limitations in current unimodal and naïve fusion strategies. More broadly, our findings suggest that Mamba-based fusion architectures can significantly advance the field of multimodal object detection, especially under visually degraded or complex conditions.

Yayun Du, Andrew Miller, M. Khalid Jawed

We explore the locomotion of soft robots in granular medium (GM) resulting from the elastic deformation of slender rods. A low-cost, rapidly fabricable robot inspired by the physiological structure of bacteria is presented. It consists of a rigid head, with a motor and batteries embedded, and multiple elastic rods (our model for flagella) to investigate locomotion in GM. The elastic flagella are rotated at one end by the motor, and they deform due to the drag from GM, propelling the robot. The external drag is determined by the flagellar shape, while the latter changes due to the competition between external loading and elastic forces. In this coupled fluid-structure interaction problem, we observe that increasing the number of flagella can decrease or increase the propulsive speed of the robot, depending on the physical parameters of the system. This nonlinearity in the functional relation between propulsion and the parameters of this simple robot motivates us to fundamentally analyze its mechanics using theory, numerical simulation, and experiments. We present a simple Euler-Bernoulli beam theory-based analytical framework that is capable of qualitatively capturing both cases. Theoretical prediction quantitatively matches experiments when the flagellar deformation is small. To account for the geometrically nonlinear deformation often encountered in soft robots and microbes, we implement a simulation framework that incorporates discrete differential geometry-based simulations of elastic rods, a resistive force theory-based model for drag, and a modified Stokes law for the hydrodynamics of the robot head. Comparison with experimental data indicates that the simulations can quantitatively predict robotic motion. Overall, the theoretical and numerical tools presented in this paper can shed light on the design and control of this class of articulated robots in granular or fluid media.

Dezhong Tong, Andy Borum, M. Khalid Jawed

Experimental analysis of the mechanics of a deformable object, and particularly its stability, requires repetitive testing and, depending on the complexity of the object's shape, a testing setup that can manipulate many degrees of freedom at the object's boundary. Motivated by recent advancements in robotic manipulation of deformable objects, this paper addresses these challenges by constructing a method for automated stability testing of a slender elastic rod -- a canonical example of a deformable object -- using a robotic system. We focus on rod configurations with helical centerlines since the stability of a helical rod can be described using only three parameters, but experimentally determining the stability requires manipulation of both the position and orientation at one end of the rod, which is not possible using traditional experimental methods that only actuate a limited number of degrees of freedom. Using a recent geometric characterization of stability for helical rods, we construct and implement a manipulation scheme to explore the space of stable helices, and we use a vision system to detect the onset of instabilities within this space. The experimental results obtained by our automated testing system show good agreement with numerical simulations of elastic rods in helical configurations. The methods described in this paper lay the groundwork for automation to grow within the field of experimental mechanics.

Yayun Du, Bhrugu Mallajosyula, Deming Sun et al.

Modern herbicide application in agricultural settings typically relies on either large scale sprayers that dispense herbicide over crops and weeds alike or portable sprayers that require labor intensive manual operation. The former method results in overuse of herbicide and reduction in crop yield while the latter is often untenable in large scale operations. This paper presents the first fully autonomous robot for weed management for row crops capable of computer vision based navigation, weed detection, complete field coverage, and automatic recharge for under \$400. The target application is autonomous inter-row weed control in crop fields, e.g. flax and canola, where the spacing between croplines is as small as one foot. The proposed robot is small enough to pass between croplines at all stages of plant growth while detecting weeds and spraying herbicide. A recharging system incorporates newly designed robotic hardware, a ramp, a robotic charging arm, and a mobile charging station. An integrated vision algorithm is employed to assist with charger alignment effectively. Combined, they enable the robot to work continuously in the field without access to electricity. In addition, a color-based contour algorithm combined with preprocessing techniques is applied for robust navigation relying on the input from the onboard monocular camera. Incorporating such compact robots into farms could help automate weed control, even during late stages of growth, and reduce herbicide use by targeting weeds with precision. The robotic platform is field-tested in the flaxseed fields of North Dakota.

Sangmin Lim, Achyuta Yadunandan, Mohammad Khalid Jawed

The bundling of flagella is known to create a "run" phase, where the bacteria moves in a nearly straight line rather than making changes in direction. Historically, mechanical explanations for the bundling phenomenon intrigued many researchers, and significant advances were made in physical models and experimental methods. Contributing to the field of research, we present a bacteria-inspired centimeter-scale soft robotic hardware platform and a computational framework for a physically plausible simulation model of the multi-flagellated robot under low Reynolds number (~0.1). The fluid-structure interaction simulation couples the Discrete Elastic Rods algorithm with the method of Regularized Stokeslet Segments. Contact between two flagella is handled by a penalty-based method. We present a comparison between our experimental and simulation results and verify that the simulation tool can capture the essential physics of this problem. Preliminary findings on robustness to buckling provided by the bundling phenomenon and the efficiency of a multi-flagellated soft robot are compared with the single-flagellated counterparts. Observations were made on the coupling between geometry and elasticity, which manifests itself in the propulsion of the robot by nonlinear dependency on the rotational speed of the flagella.

Yayun Du, Andrew Miller, Mohammad Khalid Jawed

A wide range of microorganisms, e.g. bacteria, propel themselves by rotation of soft helical tails, also known as flagella. Due to the small size of these organisms, viscous forces overwhelm inertial effects and the flow is at low Reynolds number. In this fluid-structure problem, a competition between elastic forces and hydrodynamic (viscous) forces leads to a net propulsive force forward. A thorough understanding of this highly coupled fluid-structure interaction problem can not only help us better understand biological propulsion but also help us design bio-inspired functional robots with applications in oil spill cleanup, water quality monitoring, and infrastructure inspection. Here, we introduce arguably the simplest soft robot with a single binary control signal, which is capable of moving along an arbitrary 2D trajectory near air-fluid interface and at the interface between two fluids. The robot exploits the variation in viscosity to move along the prescribed trajectory. Our analysis of this newly introduced soft robot consists of three main components. First, we fabricate this simple robot and use it as an experimental testbed. Second, a discrete differential geometry-based modeling framework is used for simulation of the robot. Upon validation of the simulation tool, the third part of this study employs the simulations to develop a control scheme with a single binary input to make the robot follow any prescribed path.

Yayun Du, Jacqueline Lam, Karunesh Sachanandani et al.

Soft robots, in contrast to their rigid counter parts, have infinite degrees of freedom that are coupled with their interaction with the environment. We consider the locomotion of an untethered robot, in the granular medium, comprised of multiple flexible flagella that rotate about an axis by a motor. Drag from the grains causes the flagella to deform and the deformed shape generates a net forward propulsion. This external drag force depends on the shape of the flagella, while the change in flagellar shape is the result of the competition between the external loading and elastic forces. We introduce a numerical tool that couples discrete differential geometry based simulation of elastic rods - our model for flagella - and a resistive force theory based model for the drag. In parallel with simulations, we conduct experiments to quantify the propulsive speed of this class of robots. We find reasonable quantitative agreement between experiments and simulations. Owing to a rod-based kinematic representation of the robot, the simulation runs faster than real-time, and, therefore, we can use it as a design tool for this class of soft robots. We find that there is an optimal rotational speed at which maximum efficiency is achieved. Moreover, both experiments and simulations show that increasing the number of flagella decreases the speed of the robot. We also gain insight into the mechanics of granular medium - while resistive force theory can successfully describe the propulsion at low number of flagella, it fails when more flagella are added to the robot.

Mojtaba Forghani, Weicheng Huang, M. Khalid Jawed

In this paper, we analyze the inverse dynamics and control of a bacteria-inspired uniflagellar robot in a fluid medium at low Reynolds number. Inspired by the mechanism behind the locomotion of flagellated bacteria, we consider a robot comprised of a flagellum -- a flexible helical filament -- attached to a spherical head. The flagellum rotates about the head at a controlled angular velocity and generates a propulsive force that moves the robot forward. When the angular velocity exceeds a threshold value, the hydrodynamic force exerted by the fluid can cause the soft flagellum to buckle, characterized by a dramatic change in shape. In this computational study, a fluid-structure interaction model that combines Discrete Elastic Rods (DER) algorithm with Lighthill's Slender Body Theory (LSBT) is employed to simulate the locomotion and deformation of the robot. We demonstrate that the robot can follow a prescribed path in three dimensional space by exploiting buckling of the flagellum. The control scheme involves only a single (binary) scalar input -- the angular velocity of the flagellum. By triggering the buckling instability at the right moment, the robot can follow an arbitrary path in three dimensional space. We also show that the complexity of the dynamics of the helical filament can be captured using a deep neural network, from which we identify the input-output functional relationship between the control inputs and the trajectory of the robot. Furthermore, our study underscores the potential role of buckling in the locomotion of natural bacteria.