Non-stationary Delayed Combinatorial Semi-Bandit with Causally Related Rewards
This addresses the challenge of delayed feedback and non-stationarity in combinatorial bandits for applications like epidemic modeling, but it is incremental as it builds on existing bandit frameworks with causal modeling.
The paper tackles the problem of sequential decision-making with delayed feedback in non-stationary environments where rewards are causally related, developing a policy that learns structural dependencies to optimize decisions and proving a regret bound, with evaluation on synthetic and real-world datasets for Covid-19 spread detection in Italy.
Sequential decision-making under uncertainty is often associated with long feedback delays. Such delays degrade the performance of the learning agent in identifying a subset of arms with the optimal collective reward in the long run. This problem becomes significantly challenging in a non-stationary environment with structural dependencies amongst the reward distributions associated with the arms. Therefore, besides adapting to delays and environmental changes, learning the causal relations alleviates the adverse effects of feedback delay on the decision-making process. We formalize the described setting as a non-stationary and delayed combinatorial semi-bandit problem with causally related rewards. We model the causal relations by a directed graph in a stationary structural equation model. The agent maximizes the long-term average payoff, defined as a linear function of the base arms' rewards. We develop a policy that learns the structural dependencies from delayed feedback and utilizes that to optimize the decision-making while adapting to drifts. We prove a regret bound for the performance of the proposed algorithm. Besides, we evaluate our method via numerical analysis using synthetic and real-world datasets to detect the regions that contribute the most to the spread of Covid-19 in Italy.