Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations
This provides a faster surrogate modeling method for physical systems, though it appears incremental as it builds on prior-data fitted networks with a modified attention mechanism.
The paper tackles the limited effectiveness of prior-data fitted networks (PFNs) on high-dimensional regression tasks by introducing Decoupled-Value Attention (DVA), which reduces validation loss by over 50% in 5D and 10D cases and achieves mean absolute error of 1E-3 for 64D power flow equations while being 80x faster than exact Gaussian Process inference.
Prior-data fitted networks (PFNs) are a promising alternative to time-consuming Gaussian Process (GP) inference for creating fast surrogates of physical systems. PFN reduces the computational burden of GP-training by replacing Bayesian inference in GP with a single forward pass of a learned prediction model. However, with standard Transformer attention, PFNs show limited effectiveness on high-dimensional regression tasks. We introduce Decoupled-Value Attention (DVA)-- motivated by the GP property that the function space is fully characterized by the kernel over inputs and the predictive mean is a weighted sum of training targets. DVA computes similarities from inputs only and propagates labels solely through values. Thus, the proposed DVA mirrors the Gaussian-process update while remaining kernel-free. We demonstrate that the crucial factor for scaling PFNs is the attention rule rather than the architecture itself. Specifically, our results demonstrate that (a) localized attention consistently reduces out-of-sample validation loss in PFNs across different dimensional settings, with validation loss reduced by more than 50% in five- and ten-dimensional cases, and (b) the role of attention is more decisive than the choice of backbone architecture, showing that CNN-based PFNs can perform at par with their Transformer-based counterparts. The proposed PFNs provide 64-dimensional power flow equation approximations with a mean absolute error of the order of 1E-3, while being over 80x faster than exact GP inference.