Exploring Maximum Entropy Distributions with Evolutionary Algorithms
This work provides a flexible numerical method for statisticians and machine learning practitioners dealing with variational calculus problems, though it is incremental as it applies existing evolutionary strategies to a known bottleneck.
The paper tackles the problem of finding maximum entropy probability distributions for given constraints using evolutionary algorithms, demonstrating that these algorithms can approximate known distributions and handle cases without closed-form solutions.
This paper shows how to evolve numerically the maximum entropy probability distributions for a given set of constraints, which is a variational calculus problem. An evolutionary algorithm can obtain approximations to some well-known analytical results, but is even more flexible and can find distributions for which a closed formula cannot be readily stated. The numerical approach handles distributions over finite intervals. We show that there are two ways of conducting the procedure: by direct optimization of the Lagrangian of the constrained problem, or by optimizing the entropy among the subset of distributions which fulfill the constraints. An incremental evolutionary strategy easily obtains the uniform, the exponential, the Gaussian, the log-normal, the Laplace, among other distributions, once the constrained problem is solved with any of the two methods. Solutions for mixed ("chimera") distributions can be also found. We explain why many of the distributions are symmetrical and continuous, but some are not.