Representing Unordered Data Using Complex-Weighted Multiset Automata
This work provides a theoretical framework for handling unordered data in neural networks, which is incremental but offers new insights for researchers in machine learning and related fields.
The authors tackled the problem of representing unordered, variable-sized data by proposing complex-weighted multiset automata, showing that existing neural architectures like Transformers and DeepSets are special cases of their framework. They extended DeepSets to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks.
Unordered, variable-sized inputs arise in many settings across multiple fields. The ability for set- and multiset-oriented neural networks to handle this type of input has been the focus of much work in recent years. We propose to represent multisets using complex-weighted multiset automata and show how the multiset representations of certain existing neural architectures can be viewed as special cases of ours. Namely, (1) we provide a new theoretical and intuitive justification for the Transformer model's representation of positions using sinusoidal functions, and (2) we extend the DeepSets model to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks.