Structural Disentanglement in Bilinear MLPs via Architectural Inductive Bias
This work addresses the problem of model editability and generalization for researchers and practitioners in machine learning, offering a foundational insight into representational structure, though it is incremental in exploring architectural biases rather than introducing a new paradigm.
The paper tackled the problem of selective unlearning and long-horizon extrapolation in neural networks by investigating how architectural inductive biases, specifically multiplicative interactions in Bilinear MLPs, aid in structural disentanglement. It showed that bilinear parameterizations enable a 'non-mixing' property under gradient flow, allowing recovery of true operators aligned with underlying algebraic structures in experiments on modular arithmetic, cyclic reasoning, Lie group dynamics, and unlearning benchmarks.
Selective unlearning and long-horizon extrapolation remain fragile in modern neural networks, even when tasks have underlying algebraic structure. In this work, we argue that these failures arise not solely from optimization or unlearning algorithms, but from how models structure their internal representations during training. We explore if having explicit multiplicative interactions as an architectural inductive bias helps in structural disentanglement, through Bilinear MLPs. We show analytically that bilinear parameterizations possess a `non-mixing' property under gradient flow conditions, where functional components separate into orthogonal subspace representations. This provides a mathematical foundation for surgical model modification. We validate this hypothesis through a series of controlled experiments spanning modular arithmetic, cyclic reasoning, Lie group dynamics, and targeted unlearning benchmarks. Unlike pointwise nonlinear networks, multiplicative architectures are able to recover true operators aligned with the underlying algebraic structure. Our results suggest that model editability and generalization are constrained by representational structure, and that architectural inductive bias plays a central role in enabling reliable unlearning.