Dynamics-Aligned Shared Hypernetworks for Zero-Shot Actuator Inversion
This work addresses a core problem in reinforcement learning for robotics or control systems where latent contexts cause failures, but it is incremental as it builds on existing hypernetwork and context-aware methods.
The paper tackles the challenge of zero-shot generalization in contextual reinforcement learning, specifically for actuator inversion where identical actions have opposite effects under latent binary contexts, and introduces DMA*-SH, a hypernetwork framework that achieves 111.8% improvement over domain randomization and 16.1% over a baseline on a new benchmark.
Zero-shot generalization in contextual reinforcement learning remains a core challenge, particularly when the context is latent and must be inferred from data. A canonical failure mode is actuator inversion, where identical actions produce opposite physical effects under a latent binary context. We propose DMA*-SH, a framework where a single hypernetwork, trained solely via dynamics prediction, generates a small set of adapter weights shared across the dynamics model, policy, and action-value function. This shared modulation imparts an inductive bias matched to actuator inversion, while input/output normalization and random input masking stabilize context inference, promoting directionally concentrated representations. We provide theoretical support via an expressivity separation result for hypernetwork modulation, and a variance decomposition with policy-gradient variance bounds that formalize how within-mode compression improves learning under actuator inversion. For evaluation, we introduce the Actuator Inversion Benchmark (AIB), a suite of environments designed to isolate discontinuous context-to-dynamics interactions. On AIB's held-out actuator-inversion tasks, DMA*-SH achieves zero-shot generalization, outperforming domain randomization by 111.8% and surpassing a standard context-aware baseline by 16.1%.