Will Trojak

Joe Watson, Chen Song, Oliver Weeger et al. · cambridge

This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecast, with a focus on partial differential equations. These methods have attracted significant interest due to their potential impact on advancing scientific research and industrial practices by improving predictive models with small- or large-scale datasets and expressive predictive models with useful inductive biases. The survey has two parts. The first considers incorporating physics knowledge on an architectural level through objective functions, structured predictive models, and data augmentation. The second considers data as physics knowledge, which motivates looking at multi-task, meta, and contextual learning as an alternative approach to incorporating physics knowledge in a data-driven fashion. Finally, we also provide an industrial perspective on the application of these methods and a survey of the open-source ecosystem for physics-informed machine learning.

Johannes Schmude, Sujit Roy, Will Trojak et al.

Triggered by the realization that AI emulators can rival the performance of traditional numerical weather prediction models running on HPC systems, there is now an increasing number of large AI models that address use cases such as forecasting, downscaling, or nowcasting. While the parallel developments in the AI literature focus on foundation models -- models that can be effectively tuned to address multiple, different use cases -- the developments on the weather and climate side largely focus on single-use cases with particular emphasis on mid-range forecasting. We close this gap by introducing Prithvi WxC, a 2.3 billion parameter foundation model developed using 160 variables from the Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2). Prithvi WxC employs an encoder-decoder-based architecture, incorporating concepts from various recent transformer models to effectively capture both regional and global dependencies in the input data. The model has been designed to accommodate large token counts to model weather phenomena in different topologies at fine resolutions. Furthermore, it is trained with a mixed objective that combines the paradigms of masked reconstruction with forecasting. We test the model on a set of challenging downstream tasks namely: Autoregressive rollout forecasting, Downscaling, Gravity wave flux parameterization, and Extreme events estimation. The pretrained model with 2.3 billion parameters, along with the associated fine-tuning workflows, has been publicly released as an open-source contribution via Hugging Face.

Will Trojak, Rob Watson, Ashley Scillitoe et al.

Theoretical methods are developed to understand the effect of non-uniform grids on Flux Reconstruction (FR) in multi-dimensions. The analysis reveals that the same effect of expanding and contracting grids is seen in two dimensions as in one dimension. Namely, that expansions cause instability and contractions cause excess dissipation. Subsequent numerical experiments on the Taylor-Green Vortex with jittered elements show the effect of localised regions of expansion and contraction, with an initial increase in the kinetic energy observed on non-uniform meshes. Some comparison is made between second-order FR and second-order finite volume (FV). FR is found to be more resilient to mesh deformation, however, FV is found to be more resolved when operated at second order on the same mesh. In both cases, it is recommended that a kinetic energy preserving/conservation formulation should be used as this can greatly increase resilience to mesh deformation.

Will Trojak, Rob Watson, Paul G. Tucker

Filtering is often used in Large Eddy Simulation with a global filter width, instead here a filter width in the reference domain of high order Flux Reconstruction is considered. It is shown via Von Neumann analysis how filtering effects the dispersion and dissipation of the scheme when spatially and temporally discretised. With it being shown that filtering stabilises the scheme temporally by upto $25\%$ for forth order FR. The impact of filtering on error production is calculated, highlighting the reduction in convective velocity caused and showing numerically the impact on order of accuracy. Finally, the turbulent Taylor-Green case is used to understand the effect of reference domain filtering on the transition to turbulence, and a filter Reynolds number is defined that is shown to be useful in understanding the effect of filtering on simulations.

Will Trojak, Rob Watson, Paul G. Tucker

High-order CFD is gathering a broadening interest as a future industrial tool, with one such approach being Flux Reconstruction (FR). However, due to the need to mesh complex geometries if FR is to displace current, lower order methods, FR will likely have to be applied to stretched and warped meshes. Therefore, it is proposed that the analytical and numerical behaviour of FR on deformed meshes for both the 1D linear advection and the 2D Euler equations is investigated. The analytical foundation of this work is based on a modified von Neumann analysis for linearly deformed grids that is presented. The temporal stability limits for linear advection on such grids are also explored analytically and numerically, with CFL limits set out for several Runge-Kutta schemes, with the primary trend being that contracting mesh regions give rise to higher CFL limits whereas expansion leads to lower CFL limits. Lastly, the benchmarks of FR are compared to finite difference and finite volumes schemes, as are common in industry, with the comparison showing the increased wave propagating ability on warped and stretched meshes, and hence, FR;s increased resilience to mesh deformation.

Will Trojak

A new set of symmetric correction functions is presented for high-order flux reconstruction, that expands upon, while incorporating, all previous correction function sets and opens the possibility for improved performance. By considering FR applied to the linear advection equation, and through modification to the Sobolev norm, criteria are presented for a wider set of correction functions. Legendre polynomials are then used to fulfil these criterion and realise functions for third to fifth order FR. The sufficient conditions for the existence of the modified norms are also explored, before Fourier and Von Neumann analysis are applied to analytically find temporal stability limits for various Runge-Kutta temporal integration schemes. For all cases, correction functions are found that extend the temporal stability of FR. Two application-inspired investigations are performed that aim to explore the effect of aliasing and non-linear equations. In both cases unique correction functions could be found that give good performance, compared to previous FR schemes, while also improving upon the temporal stability limit.

Nayara Fonseca, Veronica Guidetti, Will Trojak

A novel comparison is presented of the effect of optimiser choice on the accuracy of physics-informed neural networks (PINNs). To give insight into why some optimisers are better, a new approach is proposed that tracks the training trajectory curvature and can be evaluated on the fly at a low computational cost. The linear advection equation is studied for several advective velocities, and we show that the optimiser choice substantially impacts PINNs model performance and accuracy. Furthermore, using the curvature measure, we found a negative correlation between the convergence error and the curvature in the optimiser local reference frame. It is concluded that, in this case, larger local curvature values result in better solutions. Consequently, optimisation of PINNs is made more difficult as minima are in highly curved regions.

Will Trojak

A unique set of correction functions for Flux Reconstruction is presented, with there derivation stemming from proving the existence of energy stability in the Lebesgue norm. The set is shown to be incredibly arbitrary with the only union to existing correction function sets being show to be for DG. Von Neumann analysis of both advection and coupled advection-diffusion is used to show that once coupled to a temporal integration method, good CFL performance can be achieved and the correction function may have better dispersion and dissipation for application to implicit LES. Lastly, the turbulent Taylor-Green vortex test case is then used to show that correction functions can be found that improve the accuracy of the scheme when compared to the error levels of Discontinuous Galerkin.