Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products
This work addresses a foundational problem in AI by proposing a distributed approach to compositional representations, potentially enhancing alignment with deep learning principles, though it appears incremental as an extension of existing tensor product methods.
The paper tackled the tension between symbolic compositional representations and continuous distributed deep learning by introducing Soft Tensor Product Representation (Soft TPR) and a corresponding autoencoder, which outperformed conventional disentanglement methods in visual representation learning, achieving state-of-the-art disentanglement and improved convergence and sample efficiency.
Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explicitly compositional representations; however, it relies on a fundamentally symbolic, concatenative representation of compositional structure that clashes with the continuous, distributed foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce Soft TPR, a representational form that encodes compositional structure in an inherently distributed, flexible manner, along with Soft TPR Autoencoder, a theoretically-principled architecture designed specifically to learn Soft TPRs. Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives -- achieving state-of-the-art disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in downstream tasks. These findings highlight the promise of a distributed and flexible approach to representing compositional structure by potentially enhancing alignment with the core principles of deep learning over the conventional symbolic approach.