Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment
This work addresses efficiency challenges in educational technology by reducing storage and maintenance for automatic scoring systems, though it appears incremental as it builds on existing model merging techniques with a specific distance metric.
The paper tackled the problem of deploying separate neural networks for automatic scoring of student responses, which increases storage and computational demands, by introducing the Gromov-Wasserstein Scoring Model Merging (GW-SMM) method to merge models based on feature distribution similarities, resulting in a unified feature extractor that outperformed human experts and a GPT-o1-based method with statistically significant improvements in micro F1 and per-label accuracy while reducing storage by half.
Automatic scoring of student responses enhances efficiency in education, but deploying a separate neural network for each task increases storage demands, maintenance efforts, and redundant computations. To address these challenges, this paper introduces the Gromov-Wasserstein Scoring Model Merging (GW-SMM) method, which merges models based on feature distribution similarities measured via the Gromov-Wasserstein distance. Our approach begins by extracting features from student responses using individual models, capturing both item-specific context and unique learned representations. The Gromov-Wasserstein distance then quantifies the similarity between these feature distributions, identifying the most compatible models for merging. Models exhibiting the smallest pairwise distances, typically in pairs or trios, are merged by combining only the shared layers preceding the classification head. This strategy results in a unified feature extractor while preserving separate classification heads for item-specific scoring. We validated our approach against human expert knowledge and a GPT-o1-based merging method. GW-SMM consistently outperformed both, achieving a higher micro F1 score, macro F1 score, exact match accuracy, and per-label accuracy. The improvements in micro F1 and per-label accuracy were statistically significant compared to GPT-o1-based merging (p=0.04, p=0.01). Additionally, GW-SMM reduced storage requirements by half without compromising much accuracy, demonstrating its computational efficiency alongside reliable scoring performance.