Herding Dynamic Weights for Partially Observed Random Field Models
This addresses the challenge of parameter learning in random field models for researchers in machine learning, offering a deterministic alternative to traditional methods, though it appears incremental as it builds on existing dynamical systems approaches.
The paper tackles the intractable problem of learning parameters in partially observable random field models by treating parameters as dynamical quantities and introducing a deterministic algorithm that generates complex dynamics for parameters and state vectors, showing that under certain conditions, trajectory averages converge to data averages without requiring expensive exponentiation operations.
Learning the parameters of a (potentially partially observable) random field model is intractable in general. Instead of focussing on a single optimal parameter value we propose to treat parameters as dynamical quantities. We introduce an algorithm to generate complex dynamics for parameters and (both visible and hidden) state vectors. We show that under certain conditions averages computed over trajectories of the proposed dynamical system converge to averages computed over the data. Our "herding dynamics" does not require expensive operations such as exponentiation and is fully deterministic.