Artem Kaznatcheev

Artem Kaznatcheev, Konrad Paul Kording

Deep Learning (DL) is a surprisingly successful branch of machine learning. The success of DL is usually explained by focusing analysis on a particular recent algorithm and its traits. Instead, we propose that an explanation of the success of DL must look at the population of all algorithms in the field and how they have evolved over time. We argue that cultural evolution is a useful framework to explain the success of DL. In analogy to biology, we use `development' to mean the process converting the pseudocode or text description of an algorithm into a fully trained model. This includes writing the programming code, compiling and running the program, and training the model. If all parts of the process don't align well then the resultant model will be useless (if the code runs at all!). This is a constraint. A core component of evolutionary developmental biology is the concept of deconstraints -- these are modification to the developmental process that avoid complete failure by automatically accommodating changes in other components. We suggest that many important innovations in DL, from neural networks themselves to hyperparameter optimization and AutoGrad, can be seen as developmental deconstraints. These deconstraints can be very helpful to both the particular algorithm in how it handles challenges in implementation and the overall field of DL in how easy it is for new ideas to be generated. We highlight how our perspective can both advance DL and lead to new insights for evolutionary biology.

Artem Kaznatcheev, Willemijn Volgering

Many combinatorial optimization problems can be formulated as finding an assignment that maximizes some pseudo-Boolean function (that we call the fitness function). Strict local search starts with some assignment and follows some update rule to proceed to an adjacent assignment of strictly higher fitness. This means that strict local search algorithms follow ascents in the fitness landscape of the pseudo-Boolean function. The complexity of the pseudo-Boolean function (and the fitness landscapes that it represents) can be parameterized by properties of the valued constraint satisfaction problem (VCSP) that encodes the pseudo-Boolean function. We focus on properties of the constraint graphs of the VCSP, with the intuition that spare graphs are less complex than dense ones. Specifically, we argue that pathwidth is the natural sparsity parameter for understanding limits on the power of strict local search. We show that prior constructions of sparse VCSPs where all ascents are exponentially long had pathwidth greater than or equal to four. We improve this this with our controlled doubling construction: a valued constraint satisfaction problem of pathwidth three where all ascents are exponentially long from a designated initial assignment. We conclude that all strict local search algorithms can be forced to take an exponential number of steps even on simple valued constraint graphs of pathwidth three.

David A. Cohen, Peter G. Jeavons, Artem Kaznatcheev et al.

Local search in combinatorial optimisation can be viewed as an uphill climb on a corresponding fitness landscape, where the assignments visited by a strict local search follow an ascent in the landscape. This hill-climbing is sometimes surprisingly efficient, but not always. Since fitness landscapes can be succinctly represented by valued constraint satisfaction problems (VCSPs), it is natural to ask: what properties of VCSPs ensure that all ascents are polynomial? Or alternatively, what are the ``simplest'' VCSPs with exponential ascents? Prior examples of VCSPs with long ascents were built up as a chain of gadgets of constraints. Here we give a simpler star of gadgets construction by gluing 2n triangles of constraints at a common centre variable. We obtain a binary VCSP on 4n + 1 Boolean variables with an exponential ascent of length 10*2^n - 9. The variable at the centre of our construction intertwines two sublandscapes with only linear ascents into one with exponential ascents. The VCSP that we construct is significantly simpler than prior constructions in terms of treedepth (reducing Ω(log n) to 3) and feedback vertex set number (reducing Ω(n) to 1). We discuss the consequences of this simplicity for the parameterized complexity of local search.

David A. Cohen, Martin C. Cooper, Artem Kaznatcheev et al.

We investigate the complexity of local search based on steepest ascent. We show that even when all variables have domains of size two and the underlying constraint graph of variable interactions has bounded treewidth (in our construction, treewidth 7), there are fitness landscapes for which an exponential number of steps may be required to reach a local optimum. This is an improvement on prior recursive constructions of long steepest ascents, which we prove to need constraint graphs of unbounded treewidth.

Artem Kaznatcheev

A fitness landscape is a genetic space -- with two genotypes adjacent if they differ in a single locus -- and a fitness function. Evolutionary dynamics produce a flow on this landscape from lower fitness to higher; reaching equilibrium only if a local fitness peak is found. I use computational complexity to question the common assumption that evolution on static fitness landscapes can quickly reach a local fitness peak. I do this by showing that the popular NK model of rugged fitness landscapes is PLS-complete for K >= 2; the reduction from Weighted 2SAT is a bijection on adaptive walks, so there are NK fitness landscapes where every adaptive path from some vertices is of exponential length. Alternatively -- under the standard complexity theoretic assumption that there are problems in PLS not solvable in polynomial time -- this means that there are no evolutionary dynamics (known, or to be discovered, and not necessarily following adaptive paths) that can converge to a local fitness peak on all NK landscapes with K = 2. Applying results from the analysis of simplex algorithms, I show that there exist single-peaked landscapes with no reciprocal sign epistasis where the expected length of an adaptive path following strong selection weak mutation dynamics is $e^{O(n^{1/3})}$ even though an adaptive path to the optimum of length less than n is available from every vertex. The technical results are written to be accessible to mathematical biologists without a computer science background, and the biological literature is summarized for the convenience of non-biologists with the aim to open a constructive dialogue between the two disciplines.