Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization
This work addresses the challenge of ensuring policy feasibility in real-world energy systems control, which is crucial for safe and efficient deployment, though it is incremental as it builds on existing neural policy methods.
The authors tackled the problem of reinforcement learning policies in energy systems violating operational constraints by proposing PROF, a method that integrates a differentiable projection layer to enforce convex constraints, resulting in a 4% improvement in energy efficiency for building operation while maintaining thermal comfort and perfectly satisfying voltage constraints in inverter control.
While reinforcement learning (RL) is gaining popularity in energy systems control, its real-world applications are limited due to the fact that the actions from learned policies may not satisfy functional requirements or be feasible for the underlying physical system. In this work, we propose PROjected Feasibility (PROF), a method to enforce convex operational constraints within neural policies. Specifically, we incorporate a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating gradients through this differentiable projection layer, making the policy cognizant of the operational constraints. We demonstrate our method on two applications: energy-efficient building operation and inverter control. In the building operation setting, we show that PROF maintains thermal comfort requirements while improving energy efficiency by 4% over state-of-the-art methods. In the inverter control setting, PROF perfectly satisfies voltage constraints on the IEEE 37-bus feeder system, as it learns to curtail as little renewable energy as possible within its safety set.