Hardware based Spatio-Temporal Neural Processing Backend for Imaging Sensors: Towards a Smart Camera
This work addresses the need for energy-efficient, compact smart cameras with advanced processing capabilities, representing a novel integration of neural networks and hardware design rather than an incremental improvement.
The paper tackles the problem of enhancing imaging sensor performance and enabling on-device processing by developing a hardware platform for cognitive imaging sensors, achieving improvements in sensitivity and signal-to-noise ratio beyond material limits through neural filtering.
In this work we show how we can build a technology platform for cognitive imaging sensors using recent advances in recurrent neural network architectures and training methods inspired from biology. We demonstrate learning and processing tasks specific to imaging sensors, including enhancement of sensitivity and signal-to-noise ratio (SNR) purely through neural filtering beyond the fundamental limits sensor materials, and inferencing and spatio-temporal pattern recognition capabilities of these networks with applications in object detection, motion tracking and prediction. We then show designs of unit hardware cells built using complementary metal-oxide semiconductor (CMOS) and emerging materials technologies for ultra-compact and energy-efficient embedded neural processors for smart cameras.