The Domain Mixed Unit: A New Neural Arithmetic Layer
This addresses the problem of improving neural networks' ability to generalize arithmetic operations, which is incremental as it builds on existing neural arithmetic unit research.
The paper introduces the Domain Mixed Unit (DMU), a neural arithmetic layer that uses a parameter gate to mix log-space and linear-space representations for addition or subtraction, achieving state-of-the-art performance on the NALM Benchmark with the highest percentage solved on multiplication and division tasks.
The Domain Mixed Unit (DMU) is a new neural arithmetic unit that learns a single parameter gate that mixes between log-space and linear-space representations while performing either addition (DMU add) or subtraction (DMU sub). Two initializations are proposed for the DMU: one covering addition and multiplication, and another covering subtraction and division. The DMU achieves state-of-the-art performance on the NALM Benchmark, a dataset designed to test the ability of neural arithmetic units to generalize arithmetic operations, specifically performing with the highest percentage solved over all seeds on multiplication and division. The DMU will be submitted as a pull request to the open-source NALM benchmark, and its code is available on GitHub at https://github.com/marict/nalm-benchmark