Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats
This addresses a specific bottleneck in agricultural monitoring by enabling detailed observation of hidden plant parts, though it is an incremental improvement combining existing robotics and modeling techniques.
The paper tackles the problem of leaf occlusion in plant phenotyping by introducing Botany-Bot, a system that uses robotics and 3D Gaussian Splat models to create annotated digital twins, achieving accuracies such as 90.8% for leaf segmentation and 77.3% for detailed imaging of occluded structures.
Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.