CoGraM: Context-sensitive granular optimization method with rollback for robust model fusion
This addresses the challenge of robust model fusion for federated and distributed learning systems, though it appears incremental as it builds on existing merging methods.
The paper tackles the problem of merging neural networks without retraining in federated and distributed learning, where common methods like weight averaging or Fisher merging often lose accuracy and are unstable. The result is CoGraM, a multi-stage, context-sensitive optimization method that significantly improves the merged network by aligning decisions with loss differences and thresholds and preventing harmful updates through rollback.
Merging neural networks without retraining is central to federated and distributed learning. Common methods such as weight averaging or Fisher merging often lose accuracy and are unstable across seeds. CoGraM (Contextual Granular Merging) is a multi-stage, context-sensitive, loss-based, and iterative optimization method across layers, neurons, and weight levels that aligns decisions with loss differences and thresholds and prevents harmful updates through rollback. CoGraM is an optimization method that addresses the weaknesses of methods such as Fisher and can significantly improve the merged network.