Learning Functions on Multiple Sets using Multi-Set Transformers
This provides a general solution for learning on multiple sets, which is important for tasks like statistical distance estimation in machine learning, though it appears incremental as it builds on existing set-based methods.
The authors tackled the problem of learning functions on multiple permutation-invariant sets by proposing a general deep architecture that is a universal approximator, demonstrating superior results on tasks like counting, alignment, and statistical distance measurements, with more accurate estimates of KL divergence and mutual information than previous specialized techniques.
We propose a general deep architecture for learning functions on multiple permutation-invariant sets. We also show how to generalize this architecture to sets of elements of any dimension by dimension equivariance. We demonstrate that our architecture is a universal approximator of these functions, and show superior results to existing methods on a variety of tasks including counting tasks, alignment tasks, distinguishability tasks and statistical distance measurements. This last task is quite important in Machine Learning. Although our approach is quite general, we demonstrate that it can generate approximate estimates of KL divergence and mutual information that are more accurate than previous techniques that are specifically designed to approximate those statistical distances.