When and Which Sensor to Observe? Timely Tracking of a Joint Markov Source
For networked monitoring systems with limited sensor access, this work provides a decision-making framework for when and which sensor to query, though the approach is incremental over existing belief-MDP and MPC techniques.
This paper addresses remote estimation of a joint Markov source using multiple sensors with heterogeneous sampling costs over an erasure channel. It proposes a monitor pull policy to minimize a weighted sum of age of incorrect information (AoII) and sampling costs, using belief-MDP and two MPC methods (MPC-WTC and RL-MPC), validated numerically.
We investigate the problem of remote estimation (at a monitor) of a discrete-time joint Markov process with individual components which can be observed with dedicated sensors. At a given time slot, the monitor has the option of staying idle or sending a pull request to one of the sensors to obtain a partial state value, while the sensors are assumed to have heterogeneous sampling costs. Our goal is to develop a monitor pull policy, i.e., determining when and towards which sensor to send a pull request, in order to minimize a weighted sum of average age of incorrect information (AoII), or in short age, and sampling costs. As the communication model, we assume an erasure channel with a fixed one-slot delay from each sensor to the monitor. In this setting, the monitor does not perfectly know either the state of the process or the age, at any given time. We first obtain a sufficient statistic, namely belief, representing the joint distribution of the age and the current state of the observed process, by using the history of all pull requests and observations. Then, we formulate the optimization problem as a continuous state-space Markov decision process (MDP), namely belief-MDP, for the solution of which we propose two model predictive control (MPC) methods, namely MPC without terminal costs (MPC-WTC), and reinforcement learning MPC (RL-MPC). The effectiveness of the proposed methods is validated by numerical examples.