Brynjulf Owren

Elena Celledoni, Davide Murari, Brynjulf Owren et al.

Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.

Elena Celledoni, Brynjulf Owren, Yajuan Sun

No Runge-Kutta method can be energy preserving for all Hamiltonian systems. But for problems in which the Hamiltonian is a polynomial, the Averaged Vector Field (AVF) method can be interpreted as a Runge-Kutta method whose weights $b_i$ and abscissae $c_i$ represent a quadrature rule of degree at least that of the Hamiltonian. We prove that when the number of stages is minimal, the Runge-Kutta scheme must in fact be identical to the AVF scheme.

Elena Celledoni, Sølve Eidnes, Brynjulf Owren et al.

The energy preserving discrete gradient methods are generalized to finite-dimensional Riemannian manifolds by definition of a discrete approximation to the Riemannian gradient, a retraction, and a coordinate center function. The resulting schemes are intrinsic and do not depend on a particular choice of coordinates, nor on embedding of the manifold in a Euclidean space. Generalizations of well-known discrete gradient methods, such as the average vector field method and the Itoh--Abe method are obtained. It is shown how methods of higher order can be constructed via a collocation-like approach. Local and global error bounds are derived in terms of the Riemannian distance function and the Levi-Civita connection. Some numerical results on spin system problems are presented.

Elena Celledoni, David McLaren, Brynjulf Owren et al.

Given a quadratic vector field on \mathbb{R}^n possessing a quadratic first integral depending on two of the independent variables, we give a constructive proof that Kahan's discretization method exactly preserves a nearby modifed integral. Building on this result, we present a family of integrable quadratic vector fields (including the Euler top) whose Kahan discretization is a novel 10-parameter family of integrable maps.

Ferdia Sherry, Elena Celledoni, Matthias J. Ehrhardt et al.

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.

Elena Celledoni, Robert I. McLachlan, David I. McLaren et al.

Applying Kahan's discretization to the reduced Nahm equations, we obtain two classes of integrable mappings.

Sølve Eidnes, Brynjulf Owren, Torbjørn Ringholm

A method for constructing first integral preserving numerical schemes for time-dependent partial differential equations on non-uniform grids is presented. The method can be used with both finite difference and partition of unity approaches, thereby also including finite element approaches. The schemes are then extended to accommodate $r$-, $h$- and $p$-adaptivity. The method is applied to the Korteweg-de Vries equation and the Sine-Gordon equation and results from numerical experiments are presented.

Elena Celledoni, Brynjulf Owren

The discrete gradient approach is generalized to yield integral preserving methods for differential equations in Lie groups.

Brynjulf Owren

In this survey we discuss a wide variety of aspects related to Lie group integrators. These numerical integration schemes for differential equations on manifolds have been studied in a general and systematic manner since the 1990s and the activity has since then branched out in several different subareas, focussing both on theoretical and practical issues. From two alternative setups, using either frames or Lie group actions on a manifold, we here introduce the most important classes of schemes used to integrate nonlinear ordinary differential equations on Lie groups and manifolds. We describe a number of different applications where there is a natural action by a Lie group on a manifold such that our integrators can be implemented. An issue which is not well understood is the role of isotropy and how it affects the behaviour of the numerical methods. The order theory of numerical Lie group integrators has become an advanced subtopic in its own right, and here we give a brief introduction on a somewhat elementary level. Finally, we shall discuss Lie group integrators having the property that they preserve a symplectic structure or a first integral.

Håkon Marthinsen, Brynjulf Owren

Symplectic integration of autonomous Hamiltonian systems is a well-known field of study in geometric numerical integration, but for non-autonomous systems the situation is less clear, since symplectic structure requires an even number of dimensions. We show that one possible extension of symplectic methods in the autonomous setting to the non-autonomous setting is obtained by using canonical transformations. Many existing methods fit into this framework. We also perform experiments which indicate that for exponential integrators, the canonical and symmetric properties are important for good long time behaviour. In particular, the theoretical and numerical results support the well documented fact from the literature that exponential integrators for non-autonomous linear problems have superior accuracy compared to general ODE schemes.

Charles Curry, Brynjulf Owren

We introduce variable stepsize commutator free Lie group integrators, where the error control is achieved using embedded Runge-Kutta pairs.

Arturo De Marinis, Davide Murari, Elena Celledoni et al.

We study the approximation properties of shallow neural networks whose activation function is defined as the flow map of a neural ordinary differential equation (neural ODE) at the final time of the integration interval. We prove the universal approximation property (UAP) of such shallow neural networks in the space of continuous functions. Furthermore, we investigate the approximation properties of shallow neural networks whose parameters satisfy specific constraints. In particular, we constrain the Lipschitz constant of the neural ODE's flow map and the norms of the weights to increase the network's stability. We prove that the UAP holds if we consider either constraint independently. When both are enforced, there is a loss of expressiveness, and we derive approximation bounds that quantify how accurately such a constrained network can approximate a continuous function.

Elena Celledoni, James Jackaman, Davide Murari et al.

As supported by abundant experimental evidence, neural networks are state-of-the-art for many approximation tasks in high-dimensional spaces. Still, there is a lack of a rigorous theoretical understanding of what they can approximate, at which cost, and at which accuracy. One network architecture of practical use, especially for approximation tasks involving images, is (residual) convolutional networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than that of fully connected neural networks. This paper deals with approximation of time sequences where each observation is a matrix. We show that with relatively small networks, we can represent exactly a class of numerical discretizations of PDEs based on the method of lines. We constructively derive these results by exploiting the connections between discrete convolution and finite difference operators. Our network architecture is inspired by those typically adopted in the approximation of time sequences. We support our theoretical results with numerical experiments simulating the linear advection, heat, and Fisher equations.

Elena Celledoni, Andrea Leone, Davide Murari et al.

Recently, there has been an increasing interest in modelling and computation of physical systems with neural networks. Hamiltonian systems are an elegant and compact formalism in classical mechanics, where the dynamics is fully determined by one scalar function, the Hamiltonian. The solution trajectories are often constrained to evolve on a submanifold of a linear vector space. In this work, we propose new approaches for the accurate approximation of the Hamiltonian function of constrained mechanical systems given sample data information of their solutions. We focus on the importance of the preservation of the constraints in the learning strategy by using both explicit Lie group integrators and other classical schemes.

Elena Celledoni, Matthias J. Ehrhardt, Christian Etmann et al.

In recent years the use of convolutional layers to encode an inductive bias (translational equivariance) in neural networks has proven to be a very fruitful idea. The successes of this approach have motivated a line of research into incorporating other symmetries into deep learning methods, in the form of group equivariant convolutional neural networks. Much of this work has been focused on roto-translational symmetry of $\mathbf R^d$, but other examples are the scaling symmetry of $\mathbf R^d$ and rotational symmetry of the sphere. In this work, we demonstrate that group equivariant convolutional operations can naturally be incorporated into learned reconstruction methods for inverse problems that are motivated by the variational regularisation approach. Indeed, if the regularisation functional is invariant under a group symmetry, the corresponding proximal operator will satisfy an equivariance property with respect to the same group symmetry. As a result of this observation, we design learned iterative methods in which the proximal operators are modelled as group equivariant convolutional neural networks. We use roto-translationally equivariant operations in the proposed methodology and apply it to the problems of low-dose computerised tomography reconstruction and subsampled magnetic resonance imaging reconstruction. The proposed methodology is demonstrated to improve the reconstruction quality of a learned reconstruction method with a little extra computational cost at training time but without any extra cost at test time.

Elena Celledoni, Matthias J. Ehrhardt, Christian Etmann et al.

Over the past few years, deep learning has risen to the foreground as a topic of massive interest, mainly as a result of successes obtained in solving large-scale image processing tasks. There are multiple challenging mathematical problems involved in applying deep learning: most deep learning methods require the solution of hard optimisation problems, and a good understanding of the tradeoff between computational effort, amount of data and model complexity is required to successfully design a deep learning approach for a given problem. A large amount of progress made in deep learning has been based on heuristic explorations, but there is a growing effort to mathematically understand the structure in existing deep learning methods and to systematically design new deep learning methods to preserve certain types of structure in deep learning. In this article, we review a number of these directions: some deep neural networks can be understood as discretisations of dynamical systems, neural networks can be designed to have desirable properties such as invertibility or group equivariance, and new algorithmic frameworks based on conformal Hamiltonian systems and Riemannian manifolds to solve the optimisation problems have been proposed. We conclude our review of each of these topics by discussing some open problems that we consider to be interesting directions for future research.

Martin Benning, Elena Celledoni, Matthias J. Ehrhardt et al.

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretisation. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. The differential equation setting lends itself to learning additional parameters such as the time discretisation. We explore this extension alongside natural constraints (e.g. time steps lie in a simplex). We compare these deep learning algorithms numerically in terms of induced flow and generalisation ability.

Elena Celledoni, Sølve Eidnes, Brynjulf Owren et al.

This paper concerns an extension of discrete gradient methods to finite-dimensional Riemannian manifolds termed discrete Riemannian gradients, and their application to dissipative ordinary differential equations. This includes Riemannian gradient flow systems which occur naturally in optimization problems. The Itoh--Abe discrete gradient is formulated and applied to gradient systems, yielding a derivative-free optimization algorithm. The algorithm is tested on two eigenvalue problems and two problems from manifold valued imaging: InSAR denoising and DTI denoising.

Morten Dahlby, Brynjulf Owren

A general procedure for constructing conservative numerical integrators for time dependent partial differential equations is presented. In particular, linearly implicit methods preserving a time discretised version of the invariant is developed for systems of partial differential equations with polynomial nonlinearities. The framework is rather general and allows for an arbitrary number of dependent and independent variables with derivatives of any order. It is proved formally that second order convergence is obtained. The procedure is applied to a test case and numerical experiments are provided.

Morten Dahlby, Brynjulf Owren

The plane wave stability properties of the conservative schemes of Besse and Fei et al. for the cubic Schrödinger equation are analysed. Although the two methods possess many of the same conservation properties, we show that their stability behaviour is very different. An energy preserving generalisation of the Fei method with improved stability is presented.