Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions
This addresses security risks for real-world agent deployments in structured settings with auditable outcomes, offering a practical method for improving robustness, though it is incremental as it builds on existing red-teaming concepts.
The paper tackled the problem of agentic systems being vulnerable to adaptive adversarial strategies in strategic economic interactions, and found that agents that performed well against static baselines became consistently exploitable under profit-optimized pressure, with learned opponents discovering probing, anchoring, and deceptive commitments without explicit instruction. The result showed that distilling exploit episodes into prompt rules made most failures ineffective and substantially improved target performance.
As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fixed library of prompt attacks to adaptive strategies that steer agents toward unfavorable outcomes. We propose profit-driven red teaming, a stress-testing protocol that replaces handcrafted attacks with a learned opponent trained to maximize its profit using only scalar outcome feedback. The protocol requires no LLM-as-judge scoring, attack labels, or attack taxonomy, and is designed for structured settings with auditable outcomes. We instantiate it in a lean arena of four canonical economic interactions, which provide a controlled testbed for adaptive exploitability. In controlled experiments, agents that appear strong against static baselines become consistently exploitable under profit-optimized pressure, and the learned opponent discovers probing, anchoring, and deceptive commitments without explicit instruction. We then distill exploit episodes into concise prompt rules for the agent, which make most previously observed failures ineffective and substantially improve target performance. These results suggest that profit-driven red-team data can provide a practical route to improving robustness in structured agent settings with auditable outcomes.