Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
This work provides a foundational mathematical framework for understanding exploration in adaptive systems, which could impact fields like machine learning and cognitive science, though it is incremental in refining existing theoretical concepts.
The paper mathematically investigates the origin of information-seeking exploration in control, showing that it arises from minimizing divergence objectives rather than utility maximization, and proposes a dichotomy between evidence and divergence objectives to better understand exploratory behavior.
The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. While many approaches have been taken, one approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' which is often implemented in terms of maximizing the agents information gain about the world -- an approach which has been widely studied in machine learning and cognitive science. In this paper we mathematically investigate the nature and meaning of such approaches and demonstrate that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between \emph{evidence} objectives, which correspond to well-known reward or utility maximizing objectives in the literature, and \emph{divergence} objectives which instead seek to minimize the divergence between the agent's expected and desired futures, and argue that this new class of divergence objectives could form the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization.