Approximating Fixpoints of Approximated Functions
This work addresses a foundational issue in quantitative semantics and verification, with incremental improvements for specific domains like reinforcement learning and stochastic games.
The paper tackles the problem of approximating the least fixpoint of monotone and non-expansive functions when only an approximating sequence is available, proposing a dampened Mann iteration scheme that guarantees convergence to the least fixpoint under suitable conditions, with applications in model-based reinforcement learning for Markov decision processes and probabilistic systems.
Fixpoints are ubiquitous in computer science and when dealing with quantitative semantics and verification one often considers least fixpoints of (higher-dimensional) functions over the non-negative reals. We show how to approximate the least fixpoint of such functions, focusing on the case in which they are not known precisely, but represented by a sequence of approximating functions that converge to them. We concentrate on monotone and non-expansive functions, for which uniqueness of fixpoints is not guaranteed and standard fixpoint iteration schemes might get stuck at a fixpoint that is not the least. Our main contribution is the identification of an iteration scheme, a variation of Mann iteration with a dampening factor, which, under suitable conditions, is shown to guarantee convergence to the least fixpoint of the function of interest. We then argue that these results are relevant in the context of model-based reinforcement learning for Markov decision processes, showing how the proposed iteration scheme instantiates and allows us to derive convergence to the optimal expected return. More generally, we show that our results can be used to iterate to the least fixpoint almost surely for systems where the function of interest can be approximated with given probabilistic error bounds, as it happens for probabilistic systems, such as simple stochastic games, which can be explored via sampling.