Efficient Implementation of Reinforcement Learning over Homomorphic Encryption
This work addresses privacy concerns for cloud-based reinforcement learning applications, though it is incremental by adapting existing RL methods to encrypted settings.
The paper tackled the challenge of implementing reinforcement learning over fully homomorphic encryption for privacy-preserving control synthesis, focusing on Relative-Entropy-regularized RL to avoid comparison operations and demonstrating convergence with acceptable approximation error in a grid world simulation.
We investigate encrypted control policy synthesis over the cloud. While encrypted control implementations have been studied previously, we focus on the less explored paradigm of privacy-preserving control synthesis, which can involve heavier computations ideal for cloud outsourcing. We classify control policy synthesis into model-based, simulator-driven, and data-driven approaches and examine their implementation over fully homomorphic encryption (FHE) for privacy enhancements. A key challenge arises from comparison operations (min or max) in standard reinforcement learning algorithms, which are difficult to execute over encrypted data. This observation motivates our focus on Relative-Entropy-regularized reinforcement learning (RL) problems, which simplifies encrypted evaluation of synthesis algorithms due to their comparison-free structures. We demonstrate how linearly solvable value iteration, path integral control, and Z-learning can be readily implemented over FHE. We conduct a case study of our approach through numerical simulations of encrypted Z-learning in a grid world environment using the CKKS encryption scheme, showing convergence with acceptable approximation error. Our work suggests the potential for secure and efficient cloud-based reinforcement learning.