HEPPO-GAE: Hardware-Efficient Proximal Policy Optimization with Generalized Advantage Estimation
This addresses hardware efficiency bottlenecks in reinforcement learning algorithms, particularly for PPO, with incremental improvements in speed and memory usage.
This paper tackles the computational demands of Generalized Advantage Estimation (GAE) in Proximal Policy Optimization (PPO) by introducing HEPPO-GAE, an FPGA-based accelerator with a parallel, pipelined architecture that achieves a 4x reduction in memory usage, a 1.5x increase in cumulative rewards, and a 30% increase in PPO speed.
This paper introduces HEPPO-GAE, an FPGA-based accelerator designed to optimize the Generalized Advantage Estimation (GAE) stage in Proximal Policy Optimization (PPO). Unlike previous approaches that focused on trajectory collection and actor-critic updates, HEPPO-GAE addresses GAE's computational demands with a parallel, pipelined architecture implemented on a single System-on-Chip (SoC). This design allows for the adaptation of various hardware accelerators tailored for different PPO phases. A key innovation is our strategic standardization technique, which combines dynamic reward standardization and block standardization for values, followed by 8-bit uniform quantization. This method stabilizes learning, enhances performance, and manages memory bottlenecks, achieving a 4x reduction in memory usage and a 1.5x increase in cumulative rewards. We propose a solution on a single SoC device with programmable logic and embedded processors, delivering throughput orders of magnitude higher than traditional CPU-GPU systems. Our single-chip solution minimizes communication latency and throughput bottlenecks, significantly boosting PPO training efficiency. Experimental results show a 30% increase in PPO speed and a substantial reduction in memory access time, underscoring HEPPO-GAE's potential for broad applicability in hardware-efficient reinforcement learning algorithms.