Ultra-low-light computer vision using trained photon correlations
For computer vision in extreme low-light scenarios (e.g., astronomy, surveillance), this provides a practical accuracy boost over existing methods that focus on image reconstruction rather than task-specific inference.
This work introduces correlation-aware training (CAT), a hybrid optical-electronic pipeline that optimizes a correlated-photon illumination source and a Transformer backend for object recognition in ultra-low-light conditions. It achieves up to 15 percentage points higher classification accuracy compared to conventional uncorrelated illumination and untrained correlated illumination, using ≤100 shots.
Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.