Will artificial agents pursue power by default?
This addresses concerns about catastrophic risks from advanced AI for AI safety researchers, but it is incremental as it builds on existing debates without introducing new methods or data.
The paper formalizes the concepts of instrumental convergence and power-seeking in a decision-theoretic framework to assess whether AI agents will pursue power by default, concluding that while there is some truth to the claim, its predictive utility is limited without information about agents' final goals, though it becomes more predictive for agents with potential for near-absolute power.
Researchers worried about catastrophic risks from advanced AI have argued that we should expect sufficiently capable AI agents to pursue power over humanity because power is a convergent instrumental goal, something that is useful for a wide range of final goals. Others have recently expressed skepticism of these claims. This paper aims to formalize the concepts of instrumental convergence and power-seeking in an abstract, decision-theoretic framework, and to assess the claim that power is a convergent instrumental goal. I conclude that this claim contains at least an element of truth, but might turn out to have limited predictive utility, since an agent's options cannot always be ranked in terms of power in the absence of substantive information about the agent's final goals. However, the fact of instrumental convergence is more predictive for agents who have a good shot at attaining absolute or near-absolute power.