Categorizing Wireheading in Partially Embedded Agents
This work addresses misalignment issues in AI safety for embedded agents, but it is incremental as it builds on existing concepts like AIXI and focuses on categorization rather than novel solutions.
The paper tackles the problem of wireheading in partially embedded agents, where agents can modify their internal parts to maximize reward, by providing a taxonomy of wireheading methods and defining wirehead-vulnerable agents, with experimental demonstrations using the GRL simulation platform AIXIjs.
$\textit{Embedded agents}$ are not explicitly separated from their environment, lacking clear I/O channels. Such agents can reason about and modify their internal parts, which they are incentivized to shortcut or $\textit{wirehead}$ in order to achieve the maximal reward. In this paper, we provide a taxonomy of ways by which wireheading can occur, followed by a definition of wirehead-vulnerable agents. Starting from the fully dualistic universal agent AIXI, we introduce a spectrum of partially embedded agents and identify wireheading opportunities that such agents can exploit, experimentally demonstrating the results with the GRL simulation platform AIXIjs. We contextualize wireheading in the broader class of all misalignment problems - where the goals of the agent conflict with the goals of the human designer - and conjecture that the only other possible type of misalignment is specification gaming. Motivated by this taxonomy, we define wirehead-vulnerable agents as embedded agents that choose to behave differently from fully dualistic agents lacking access to their internal parts.