Tensor Memory Engine: On-the-fly Data Reorganization for Ideal Locality
This work addresses the memory wall problem for data-intensive edge computing by enabling ideal locality without application redesign, but the evaluation is limited to FPGA prototypes.
The paper proposes a hardware/software co-designed Tensor Memory Engine that reorganizes data on-the-fly to provide ideal cache locality, eliminating the need for costly software redesigns or inflated memory footprints. The approach is implemented on commercial SoC/FPGA platforms and decouples computation from memory access.
The shift to data-intensive processing from the cloud to the edge has introduced new challenges and expectations for the next generation of intelligent computing systems. As the memory wall continues to grow, modern systems can only meet these performance expectations by displaying data access patterns that exhibit ideal layouts in memory and ideal spatiotemporal locality in caches. However, only a few data-intensive applications are characterized by ideal locality. Instead, most applications exhibit either (i) poor locality when naively implemented and must undergo costly redesigns and tuning or (ii) inflated memory footprint to offer proper locality. To address the aforementioned challenges, we propose a hardware/software co-designed approach that can be implemented on commercially available SoC/FPGA platforms. Our approach seamlessly inserts in the CPUs' data path a Tensor Memory Engine that provides data with an ideal cache locality to running applications by (i) accessing the memory on behalf of the CPUs and (ii) composing a re-organized view of the memory layout. Unlike in- and near-memory computing approaches, it sets itself apart by clearly decoupling computing and memory accesses; computation is still performed on CPUs while the data re-organization is delegated to the Tensor Memory Engine.