Exploring the Dimensions of a Variational Neuron
This work addresses the challenge of local probabilistic modeling in neural networks for researchers in machine learning, though it appears incremental as it builds on existing variational methods by relocating them to the neuron level.
The paper tackles the problem of modeling uncertainty at the neuron level in neural networks by introducing EVE, a variational distributional neuron, and shows that varying its internal latent dimensionality and other structural properties shapes its learned operating regime, with some neuron-level variables being measurable and related to downstream behavior in forecasting and tabular settings.
We introduce EVE (Elemental Variational Expanse), a variational distributional neuron formulated as a local probabilistic computational unit with an explicit prior, an amortized posterior, and unit-level variational regularization. In most modern architectures, uncertainty is modeled through global latent variables or parameter uncertainty, while the computational unit itself remains scalar. EVE instead relocates probabilistic structure to the neuron level, making it locally observable and controllable. In this paper, the term dimensions refers primarily to the neuron's internal latent dimensionality, denoted by k. We study how varying k, from the atomic case k = 1 to higher-dimensional latent spaces, changes the neuron's learned operating regime. We then examine how this main axis interacts with two additional structural properties: local capacity control and temporal persistence through a neuron-level autoregressive extension. To support this study, EVE is instrumented with internal diagnostics and constraints, including effective KL, a target band on mu^2, out-of-band fractions, and indicators of drift and collapse. Across selected forecasting and tabular settings, we show that latent dimensionality, control, and temporal extension shape the neuron's internal regime, and that some neuron-level variables are measurable, informative, and related to downstream behavior. Overall, the paper provides an experimentally grounded first map of the design space opened by a variational neuron.