Task-Restricted Symmetries in Recurrent Weight Space
For researchers studying recurrent networks, this work provides a diagnostic tool to identify which weight perturbations preserve trained behavior, but the findings are task- and solution-specific, not universal.
The paper studies functional redundancy in recurrent weight space of one-layer tanh RNNs using ordered real Schur coordinates. It finds that certain nonnormal Schur couplings can be removed with little loss in some trained solutions for tasks like copy, flip-flop, sine generation, and context-dependent integration, while others are necessary for accurate performance.
Recurrent networks can contain substantial functional redundancy in weight space: changing a recurrent matrix may leave the input-output rollout nearly unchanged on a task distribution, while similar-scale changes can destroy the same behavior. We study this redundancy in one-layer tanh RNNs using ordered real Schur coordinates. The Schur form separates spectral blocks from directed nonnormal couplings, giving a diagnostic basis for structured ablations that keep the input and readout maps fixed. In a fixed-length copy task, selected nonnormal Schur couplings can be removed with little loss in some trained solutions, whereas other couplings are necessary for accurate autonomous replay. Across flip-flop, sine generation, and context-dependent integration, the loss-preserving ablation profile varies across tasks and trained solutions. These results identify candidate approximate functional invariances, not universal symmetries of recurrent weight space. Schur-coordinate ablations provide a practical diagnostic for which structured perturbations preserve a trained recurrent solution and which ones disrupt its computation.