Extrapolating Volition with Recursive Information Markets
For AI alignment researchers, this work proposes a novel mechanism for scalable oversight and extrapolated volition, though the analysis remains theoretical.
The paper analyzes a mechanism using LLM buyers to overcome information asymmetry in markets, and introduces a recursive version with applications to AI alignment. Formal analysis shows it incentivizes pricing information according to its true value.
One of the impediments to the efficiency of information markets is the inherent information asymmetry present in them, exacerbated by the "buyer's inspection paradox" (the buyer cannot mitigate the asymmetry by "inspecting" the information, because in doing so the buyer obtains the information without paying for it). Previous work has suggested that using Large Language Model (LLM) buyers to inspect and purchase information could overcome this information asymmetry, as an LLM buyer can simply "forget" the information it inspects. In this work, we analyze this mechanism formally through a "value-of-information" paradigm, i.e. whether it incentivizes information to be priced and provided in accordance with its "true value". We focus in particular on our new recursive version of the mechanism, which we believe has a range of applications including in AI alignment research, where it is related to Extrapolated Volition and Scalable Oversight.