Engression: Extrapolation through the Lens of Distributional Regression
This addresses the challenging extrapolation problem in machine learning for scenarios with pre-additive noise models, offering a new generative approach, though it is incremental as it builds on existing distributional regression methods.
The paper tackles the problem of extrapolation in nonlinear regression by proposing 'engression', a neural network-based distributional regression method that models the full conditional distribution, and shows it can successfully extrapolate under certain assumptions where traditional methods fail, with empirical validation on simulated and real data.
Distributional regression aims to estimate the full conditional distribution of a target variable, given covariates. Popular methods include linear and tree-ensemble based quantile regression. We propose a neural network-based distributional regression methodology called `engression'. An engression model is generative in the sense that we can sample from the fitted conditional distribution and is also suitable for high-dimensional outcomes. Furthermore, we find that modelling the conditional distribution on training data can constrain the fitted function outside of the training support, which offers a new perspective to the challenging extrapolation problem in nonlinear regression. In particular, for `pre-additive noise' models, where noise is added to the covariates before applying a nonlinear transformation, we show that engression can successfully perform extrapolation under some assumptions such as monotonicity, whereas traditional regression approaches such as least-squares or quantile regression fall short under the same assumptions. Our empirical results, from both simulated and real data, validate the effectiveness of the engression method and indicate that the pre-additive noise model is typically suitable for many real-world scenarios. The software implementations of engression are available in both R and Python.