Deep ChArUco: Dark ChArUco Marker Pose Estimation
This addresses the problem of robust camera calibration and pose estimation for robotics and augmented reality applications in adverse conditions, but it is incremental as it builds on existing ChArUco and deep learning methods.
The paper tackled the problem of ChArUco marker pose estimation in poor lighting and motion blur conditions, and the result was a real-time system called Deep ChArUco that outperformed traditional OpenCV-based methods in these challenging scenarios.
ChArUco boards are used for camera calibration, monocular pose estimation, and pose verification in both robotics and augmented reality. Such fiducials are detectable via traditional computer vision methods (as found in OpenCV) in well-lit environments, but classical methods fail when the lighting is poor or when the image undergoes extreme motion blur. We present Deep ChArUco, a real-time pose estimation system which combines two custom deep networks, ChArUcoNet and RefineNet, with the Perspective-n-Point (PnP) algorithm to estimate the marker's 6DoF pose. ChArUcoNet is a two-headed marker-specific convolutional neural network (CNN) which jointly outputs ID-specific classifiers and 2D point locations. The 2D point locations are further refined into subpixel coordinates using RefineNet. Our networks are trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and extreme data augmentation. We evaluate Deep ChArUco in challenging low-light, high-motion, high-blur scenarios and demonstrate that our approach is superior to a traditional OpenCV-based method for ChArUco marker detection and pose estimation.