A Frequency-aware Software Cache for Large Recommendation System Embeddings
This addresses memory constraints for Internet companies training large recommendation systems, though it appears incremental as it builds on existing caching and parallel training approaches.
The authors tackled the problem of training deep learning recommendation models with embedding tables too large for GPU memory by proposing a frequency-aware software cache that dynamically manages embeddings between CPU and GPU. Their prototype system achieved decent end-to-end training speed while keeping only 1.5% of embedding parameters in GPU memory.
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies. The embedding tables of DLRMs are too large to fit on GPU memory entirely. We propose a GPU-based software cache approaches to dynamically manage the embedding table in the CPU and GPU memory space by leveraging the id's frequency statistics of the target dataset. Our proposed software cache is efficient in training entire DLRMs on GPU in a synchronized update manner. It is also scaled to multiple GPUs in combination with the widely used hybrid parallel training approaches. Evaluating our prototype system shows that we can keep only 1.5% of the embedding parameters in the GPU to obtain a decent end-to-end training speed.