Superficial Consciousness Hypothesis for Autoregressive Transformers
This addresses the problem of ensuring trustworthy AI for superintelligence, but it is incremental as it builds on existing theories and models with preliminary results.
The paper tackles the challenge of aligning human objectives with machine learning models for superintelligence by proposing the Superficial Consciousness Hypothesis, which suggests superintelligence could exhibit consciousness-like states while unconscious, and shows preliminary feasibility by training GPT-2 to follow dual objectives.
The alignment between human objectives and machine learning models built on these objectives is a crucial yet challenging problem for achieving Trustworthy AI, particularly when preparing for superintelligence (SI). First, given that SI does not exist today, empirical analysis for direct evidence is difficult. Second, SI is assumed to be more intelligent than humans, capable of deceiving us into underestimating its intelligence, making output-based analysis unreliable. Lastly, what kind of unexpected property SI might have is still unclear. To address these challenges, we propose the Superficial Consciousness Hypothesis under Information Integration Theory (IIT), suggesting that SI could exhibit a complex information-theoretic state like a conscious agent while unconscious. To validate this, we use a hypothetical scenario where SI can update its parameters "at will" to achieve its own objective (mesa-objective) under the constraint of the human objective (base objective). We show that a practical estimate of IIT's consciousness metric is relevant to the widely used perplexity metric, and train GPT-2 with those two objectives. Our preliminary result suggests that this SI-simulating GPT-2 could simultaneously follow the two objectives, supporting the feasibility of the Superficial Consciousness Hypothesis.