Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning
This work addresses the problem of maintaining safety during deployment in in-context RL for practitioners who need to adapt online without retraining, offering a practical improvement over existing methods.
The paper introduces a latent Q-Barrier shield for safe in-context reinforcement learning that improves reward-safety tradeoffs under out-of-distribution shifts without test-time parameter updates. Across five benchmarks, the shield achieves higher return in four and matches or lowers average episode cost in all five compared to a strong baseline.
Safe in-context reinforcement learning (ICRL) adapts online from interaction history without test-time parameter updates while controlling episode cost under a safety budget. Under out-of-distribution (OOD) deployment shifts, pretraining-only safe ICRL can give poor reward-safety tradeoffs because the remaining budget affects behavior only through frozen policy conditioning, not an explicit action-level check against predicted future cost. We propose a latent Q-Barrier shield that learns a context representation, latent dynamics, and an ensemble cost critic before deployment. Without parameter updates, the shield infers context from history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. We prove a conditional, error-decomposed barrier-margin result: a Q-Barrier-satisfying action leaves the next latent-budget state with an approximately budget-safe continuation under the learned critic, up to Bellman and latent-prediction errors. Across five safe ICRL benchmarks, the shield improves deployment-time reward-safety tradeoffs over a strong safe-ICRL baseline: after a short context window, it achieves higher return in four of five benchmarks while matching or lowering average episode cost in all five.