Jeffrey Humpherys

Jeffrey Humpherys, Kevin Zumbrun

As described in the classic works of Lee--Stewart and Short--Stewart, the numerical evaluation of linear stability of planar detonation waves is a computationally intensive problem of considerable interest in applications. Reexamining this problem from a modern numerical Evans function point of view, we derive a new algorithm for their stability analysis, related to a much older method of Erpenbeck, that, while equally simple and easy to implement as the standard method introduced by Lee--Stewart, appears to be potentially faster and more stable.

Chris Hettinger, Tanner Christensen, Jeffrey Humpherys et al.

Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers. We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small image classification networks. Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts. Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

Chris Hettinger, Tanner Christensen, Ben Ehlert et al.

We present a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision trees. The main idea is that layers can be trained one at a time, and once they are trained, the input data are mapped forward through the layer to create a new learning problem. The process is repeated, transforming the data through multiple layers, one at a time, rendering a new data set, which is expected to be better behaved, and on which a final output layer can achieve good performance. We call this forward thinking and demonstrate a proof of concept by achieving state-of-the-art accuracy on the MNIST dataset for convolutional neural networks. We also provide a general mathematical formulation of forward thinking that allows for other types of deep learning problems to be considered.

Kevin Miller, Chris Hettinger, Jeffrey Humpherys et al.

The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing for (i) different types of learning functions in the network, other than neurons, and (ii) the ability to adaptively deepen the network as needed to improve results. This is done by training one layer at a time, and once a layer is trained, the input data are mapped forward through the layer to create a new learning problem. The process is then repeated, transforming the data through multiple layers, one at a time, rendering a new dataset, which is expected to be better behaved, and on which a final output layer can achieve good performance. In the case where the neurons of deep neural nets are replaced with decision trees, we call the result a Forward Thinking Deep Random Forest (FTDRF). We demonstrate a proof of concept by applying FTDRF on the MNIST dataset. We also provide a general mathematical formulation that allows for other types of deep learning problems to be considered.

Jeffrey Humpherys, Kevin Zumbrun

In Evans function computations of the spectra of asymptotically constant-coefficient linear operators, a basic issue is the efficient and numerically stable computation of subspaces evolving according to the associated eigenvalue ODE. For small systems, a fast, shooting algorithm may be obtained by representing subspaces as single exterior products \cite{AS,Br.1,Br.2,BrZ,BDG}. For large systems, however, the dimension of the exterior-product space quickly becomes prohibitive, growing as $\binom{n}{k}$, where $n$ is the dimension of the system written as a first-order ODE and $k$ (typically $\sim n/2$) is the dimension of the subspace. We resolve this difficulty by the introduction of a simple polar coordinate algorithm representing ``pure'' (monomial) products as scalar multiples of orthonormal bases, for which the angular equation is a numerically optimized version of the continuous orthogonalization method of Drury--Davey \cite{Da,Dr} and the radial equation is evaluable by quadrature. Notably, the polar-coordinate method preserves the important property of analyticity with respect to parameters.