Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization
For robotic domains with conflicting objectives, this work addresses the instability of non-linear scalarization in deep RL, enabling access to non-convex Pareto front regions.
The paper proposes an Adaptive Smooth Tchebycheff framework for multi-objective reinforcement learning that dynamically adjusts optimization curvature based on gradient conflict, enabling discovery of Pareto-optimal policies in non-convex regions. On a robotic stealth visual search task, it outperforms linear and static non-linear scalarization methods.
Multi-objective reinforcement learning in robotic domains requires balancing complex, non-convex trade-offs between conflicting objectives. While linear scalarization methods provide stability, they are theoretically incapable of recovering solutions within non-convex regions of the Pareto front. Conversely, static non-linear scalarizations (e.g., Tchebycheff) can theoretically access these regions but often suffer from severe gradient variance and optimization instability in deep RL. In this work, we propose an Adaptive Smooth Tchebycheff framework that resolves this tension by dynamically modulating the curvature of the optimization landscape. We introduce a novel conflict-driven controller that regulates the optimization smoothness based on real-time gradient interference. This allows the agent to anneal toward precise, non-convex scalarization when objectives align, while elastically reverting to stable, smooth approximations when destructive gradient conflicts emerge. We validate our approach on a challenging robotic stealth visual search task -- a proxy for monitoring of protected/fragile ecosystems -- where an agent must balance search, exposure/interference minimization and exploration speed. Extensive ablations confirm that our conflict-aware adaptation enables the robust discovery of Pareto-optimal policies in non-convex regions inaccessible to linear baselines and unstable for static non-linear methods. Website: https://alejandromllo.github.io/research/pasta/