Efficient Memory Management for Deep Neural Net Inference
This work addresses memory constraints for deep neural net inference on edge devices, offering an incremental improvement over existing techniques.
The paper tackled the problem of memory inefficiency in deep neural net inference on mobile and embedded devices by exploring strategies to share memory buffers among intermediate tensors, resulting in up to 11% smaller memory footprint compared to state-of-the-art methods.
While deep neural net inference was considered a task for servers only, latest advances in technology allow the task of inference to be moved to mobile and embedded devices, desired for various reasons ranging from latency to privacy. These devices are not only limited by their compute power and battery, but also by their inferior physical memory and cache, and thus, an efficient memory manager becomes a crucial component for deep neural net inference at the edge. We explore various strategies to smartly share memory buffers among intermediate tensors in deep neural nets. Employing these can result in up to 11% smaller memory footprint than the state of the art.