Embedded Agency
This addresses foundational issues in AI and decision theory for researchers, but it is incremental as it surveys existing challenges without proposing new solutions.
The paper tackles the problem of formalizing reasoning for agents that are embedded within their environment, highlighting obstacles such as optimizing non-functional environments, using self-contained models, and self-referential reasoning, without providing specific results or numbers.
Traditional models of rational action treat the agent as though it is cleanly separated from its environment, and can act on that environment from the outside. Such agents have a known functional relationship with their environment, can model their environment in every detail, and do not need to reason about themselves or their internal parts. We provide an informal survey of obstacles to formalizing good reasoning for agents embedded in their environment. Such agents must optimize an environment that is not of type "function"; they must rely on models that fit within the modeled environment; and they must reason about themselves as just another physical system, made of parts that can be modified and that can work at cross purposes.