Learning When Not to Learn: Risk-Sensitive Abstention in Bandits with Unbounded Rewards
This addresses safety concerns for deploying learning agents in high-stakes environments, though it is incremental as it builds on prior work by removing the need for a mentor.
The paper tackles the problem of safe sequential decision-making in high-stakes AI applications where errors can cause irreparable damage, by formalizing a contextual bandit model with an abstain option and proposing a caution-based algorithm that achieves sublinear regret guarantees under i.i.d. inputs.
In high-stakes AI applications, even a single action can cause irreparable damage. However, nearly all of sequential decision-making theory assumes that all errors are recoverable (e.g., by bounding rewards). Standard bandit algorithms that explore aggressively may cause irreparable damage when this assumption fails. Some prior work avoids irreparable errors by asking for help from a mentor, but a mentor may not always be available. In this work, we formalize a model of learning with unbounded rewards without a mentor as a two-action contextual bandit with an abstain option: at each round the agent observes an input and chooses either to abstain (always 0 reward) or to commit (execute a preexisting task policy). Committing yields rewards that are upper-bounded but can be arbitrarily negative, and the commit reward is assumed Lipschitz in the input. We propose a caution-based algorithm that learns when not to learn: it chooses a trusted region and commits only where the available evidence does not already certify harm. Under these conditions and i.i.d. inputs, we establish sublinear regret guarantees, theoretically demonstrating the effectiveness of cautious exploration for deploying learning agents safely in high-stakes environments.