Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
This provides insights into interpretability for deep learning researchers, though it appears incremental as it builds on existing MoE and VAE frameworks.
The paper tackles the challenge of understanding neural network organization by exploring a Sparse Mixture of Experts Variational Autoencoder (SMoE-VAE) on the QuickDraw dataset, finding that unsupervised routing achieves superior reconstruction performance compared to supervised baselines and learns meaningful sub-categorical structures.
Understanding the internal organization of neural networks remains a fundamental challenge in deep learning interpretability. We address this challenge by exploring a novel Sparse Mixture of Experts Variational Autoencoder (SMoE-VAE) architecture. We test our model on the QuickDraw dataset, comparing unsupervised expert routing against a supervised baseline guided by ground-truth labels. Surprisingly, we find that unsupervised routing consistently achieves superior reconstruction performance. The experts learn to identify meaningful sub-categorical structures that often transcend human-defined class boundaries. Through t-SNE visualizations and reconstruction analysis, we investigate how MoE models uncover fundamental data structures that are more aligned with the model's objective than predefined labels. Furthermore, our study on the impact of dataset size provides insights into the trade-offs between data quantity and expert specialization, offering guidance for designing efficient MoE architectures.