Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp
This addresses the challenge of automating multi-pipette localization in neuroscience experiments, which is currently manual and limits scalability, though it is incremental as it builds on existing robotic and deep learning approaches.
The paper tackled the problem of precise real-time detection of multiple pipettes in robot-assisted in vivo patch-clamp, achieving over 98% accuracy within 10 μm and over 89% accuracy within 5 μm for tip localization.
In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios. To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assisted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved > 98% accuracy within 10 μm, and > 89% accuracy within 5 μm for the localisation of multi-pipette tips. The average MSE is 2.52 μm.