InterQ: A DQN Framework for Optimal Intermittent Control
This work addresses communication-control co-design for stochastic systems, offering a novel method that could improve efficiency in networked control applications, though it appears incremental as it builds on existing reinforcement learning techniques.
The paper tackles the problem of balancing communication cost and control performance in discrete-time stochastic linear systems by proposing InterQ, a deep reinforcement learning framework for optimal intermittent control, which outperforms baseline strategies like multi-period periodic scheduling and event-triggered policies in numerical evaluations.
In this letter, we explore the communication-control co-design of discrete-time stochastic linear systems through reinforcement learning. Specifically, we examine a closed-loop system involving two sequential decision-makers: a scheduler and a controller. The scheduler continuously monitors the system's state but transmits it to the controller intermittently to balance the communication cost and control performance. The controller, in turn, determines the control input based on the intermittently received information. Given the partially nested information structure, we show that the optimal control policy follows a certainty-equivalence form. Subsequently, we analyze the qualitative behavior of the scheduling policy. To develop the optimal scheduling policy, we propose InterQ, a deep reinforcement learning algorithm which uses a deep neural network to approximate the Q-function. Through extensive numerical evaluations, we analyze the scheduling landscape and further compare our approach against two baseline strategies: (a) a multi-period periodic scheduling policy, and (b) an event-triggered policy. The results demonstrate that our proposed method outperforms both baselines. The open source implementation can be found at https://github.com/AC-sh/InterQ.