HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Control
This addresses the problem of automating critical plant operations for industrial process owners, offering a solution beyond current expert capability, though it is incremental as it builds on existing offline RL methods.
The paper tackled the automation of start-up and grade-transition operations in continuous-process plants, which traditionally rely on expert operators, by introducing HOFLON, a hybrid offline learning and online optimization method. The result showed that HOFLON outperformed a leading offline-RL algorithm and achieved better cumulative rewards than the best historical transitions in two industrial case studies.
Start-ups and product grade-changes are critical steps in continuous-process plant operation, because any misstep immediately affects product quality and drives operational losses. These transitions have long relied on manual operation by a handful of expert operators, but the progressive retirement of that workforce is leaving plant owners without the tacit know-how needed to execute them consistently. In the absence of a process model, offline reinforcement learning (RL) promises to capture and even surpass human expertise by mining historical start-up and grade-change logs, yet standard offline RL struggles with distribution shift and value-overestimation whenever a learned policy ventures outside the data envelope. We introduce HOFLON (Hybrid Offline Learning + Online Optimization) to overcome those limitations. Offline, HOFLON learns (i) a latent data manifold that represents the feasible region spanned by past transitions and (ii) a long-horizon Q-critic that predicts the cumulative reward from state-action pairs. Online, it solves a one-step optimization problem that maximizes the Q-critic while penalizing deviations from the learned manifold and excessive rates of change in the manipulated variables. We test HOFLON on two industrial case studies: a polymerization reactor start-up and a paper-machine grade-change problem, and benchmark it against Implicit Q-Learning (IQL), a leading offline-RL algorithm. In both plants HOFLON not only surpasses IQL but also delivers, on average, better cumulative rewards than the best start-up or grade-change observed in the historical data, demonstrating its potential to automate transition operations beyond current expert capability.