Steven Rosenberg

Qinxun Bai, Steven Rosenberg, Wei Xu

Natural gradients have been widely studied from both theoretical and empirical perspectives, and it is commonly believed that natural gradients have advantages over standard (Euclidean) gradients in capturing the intrinsic geometric structure of the underlying function space and being invariant under reparameterization. However, for function optimization, a fundamental theoretical issue regarding the existence of natural gradients on the function space remains underexplored. We address this issue by providing a geometric perspective and mathematical framework for studying both natural gradient and standard gradient that is more complete than existing studies. The key tool that unifies natural gradient and standard gradient is a generalized form of the Neural Tangent Kernel (NTK), which we name the Generalized Tangent Kernel (GTK). Using a novel orthonormality property of GTK, we show that for a fixed parameterization, GTK determines a Riemannian metric on the entire function space which makes the standard gradient as "natural" as the natural gradient in capturing the intrinsic structure of the parameterized function space. Many aspects of this approach relate to RKHS theory. For the practical side of this theory paper, we showcase that our framework motivates new solutions to the non-immersion/degenerate case of natural gradient and leads to new families of natural/standard gradient descent methods.

Chaitanya Poolla, Abraham K. Ishihara, Dan Liddell et al.

Commercial buildings account for approximately 35% of total US electricity consumption, of which nearly two-thirds is met by fossil fuels resulting in an adverse impact on the environment. This adverse impact can be mitigated by lowering energy consumption via control of occupant plugload usage in a closed-loop building environment. In this work, we conducted multiple experiments to analyze changes in occupant plugload energy consumption due to incentives and/or visual feedback. The incentives entailed daily monetary values between $5 and $50 administered in a randomized order and the visual feedback consisted of a web-based dashboard aimed at increasing the energy awareness of participants. Experiments were performed in government office and university buildings at NASA Ames Research Park located in Moffett Field, CA. Autoregressive models were constructed to predict expected plugload savings in the presence of exogenous variables. Analysis of the data revealed modulation of plugload energy consumption can be achieved via visual feedback and incentive mechanisms suggesting that occupant-in-the-loop control architectures may be effective in the commercial building environment. Our findings indicate that the mean energy reduction due to visual feedback in office and university environments were ~9.52% and ~21.61%, respectively. By augmenting the visual feedback in the university environment with a monetary incentive, the mean energy reduction was found to be ~24.22%

Dara Gold, Steven Rosenberg

Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an approach to manifold learning where gradient descent takes place in the infinite dimensional space $\mathcal{E} = {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings $φ$ of a manifold $M$ into $\mathbb{R}^N$. Implementing a discretized version of gradient descent for $P:\mathcal{E}\to {\mathbb R}$, a penalty function that scores an embedding $φ\in \mathcal{E}$, requires estimating how far we can move in a fixed direction -- the direction of one gradient step -- before leaving the space of smooth embeddings. Our main result is to give an explicit lower bound for this step length in terms of the Riemannian geometry of $φ(M)$. In particular, we consider the case when the gradient of $P$ is pointwise normal to the embedded manifold $φ(M)$. We prove this case arises when $P$ is invariant under diffeomorphisms of $M$, a natural condition in manifold learning.

Qinxun Bai, Steven Rosenberg, Zheng Wu et al.

We study the problem of supervised learning for both binary and multiclass classification from a unified geometric perspective. In particular, we propose a geometric regularization technique to find the submanifold corresponding to a robust estimator of the class probability $P(y|\pmb{x})$. The regularization term measures the volume of this submanifold, based on the intuition that overfitting produces rapid local oscillations and hence large volume of the estimator. This technique can be applied to regularize any classification function that satisfies two requirements: firstly, an estimator of the class probability can be obtained; secondly, first and second derivatives of the class probability estimator can be calculated. In experiments, we apply our regularization technique to standard loss functions for classification, our RBF-based implementation compares favorably to widely used regularization methods for both binary and multiclass classification.