Cold-RL: Learning Cache Eviction with Offline Reinforcement Learning for NGINX
This addresses cache performance issues for web proxy operators, offering a practical RL-based solution with strict service-level objectives, though it is incremental as it builds on existing RL methods for caching.
The paper tackles the problem of inefficient cache eviction in web proxies like NGINX, which rely on least-recently-used (LRU) policies that can thrash under certain workloads. By introducing Cold-RL, a learned eviction policy using offline reinforcement learning, they achieved hit ratio improvements from 0.1436 to 0.3538 (146% gain) on a 25 MB cache and from 0.7530 to 0.8675 (15% gain) on a 100 MB cache, with minimal CPU overhead and latency within budget.
Web proxies such as NGINX commonly rely on least-recently-used (LRU) eviction, which is size agnostic and can thrash under periodic bursts and mixed object sizes. We introduce Cold-RL, a learned eviction policy for NGINX that replaces LRU's forced-expire path with a dueling Deep Q-Network served by an ONNX sidecar within a strict microsecond budget. On each eviction, Cold-RL samples the K least-recently-used objects, extracts six lightweight features (age, size, hit count, inter-arrival time, remaining TTL, and last origin RTT), and requests a bitmask of victims; a hard timeout of 500 microseconds triggers immediate fallback to native LRU. Policies are trained offline by replaying NGINX access logs through a cache simulator with a simple reward: a retained object earns one point if it is hit again before TTL expiry. We compare against LRU, LFU, size-based, adaptive LRU, and a hybrid baseline on two adversarial workloads. With a 25 MB cache, Cold-RL raises hit ratio from 0.1436 to 0.3538, a 146 percent improvement over the best classical baseline; at 100 MB, from 0.7530 to 0.8675, a 15 percent gain; and at 400 MB it matches classical methods (about 0.918). Inference adds less than 2 percent CPU overhead and keeps 95th percentile eviction latency within budget. To our knowledge, this is the first reinforcement learning eviction policy integrated into NGINX with strict SLOs.