GPUReplay: A 50-KB GPU Stack for Client ML
This enables efficient and secure GPU-accelerated ML on resource-constrained devices, though it is an incremental improvement over existing replay techniques.
GPUReplay tackles the high complexity of GPU stacks for mobile/embedded devices by recording GPU executions ahead of time and replaying them at runtime, resulting in a 50 KB executable that speeds up startup by up to 100x and works with 33 neural network implementations.
GPUReplay (GR) is a novel way for deploying GPU-accelerated computation on mobile and embedded devices. It addresses high complexity of a modern GPU stack for deployment ease and security. The idea is to record GPU executions on the full GPU stack ahead of time and replay the executions on new input at run time. We address key challenges towards making GR feasible, sound, and practical to use. The resultant replayer is a drop-in replacement of the original GPU stack. It is tiny (50 KB of executable), robust (replaying long executions without divergence), portable (running in a commodity OS, in TEE, and baremetal), and quick to launch (speeding up startup by up to two orders of magnitude). We show that GPUReplay works with a variety of integrated GPU hardware, GPU APIs, ML frameworks, and 33 neural network (NN) implementations for inference or training. The code is available at https://github.com/bakhi/GPUReplay.