Embracing the chaos: analysis and diagnosis of numerical instability in variational flows
This addresses the problem of ensuring reliability in variational flows for practitioners, though it is incremental in applying shadowing theory to this context.
The paper investigates how numerical instability affects the reliability of variational flows in sampling, density evaluation, and ELBO estimation, finding that results can remain accurate despite serious instability, and provides theoretical guarantees and a diagnostic procedure to validate these results.
In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map -- which affects sampling -- and the numerical inverse flow map does not accurately recover the initial input -- which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.